The advent of cluster computing has resulted in a thrust towards providing software mechanisms for reliability on clusters. The prevalent model for such mechanisms is to take a snapshot of the state of an application, called a checkpoint and commit it to stable storage. This checkpoint has sufficient meta-data, so that if the application fails, it can be restarted from the checkpoint. This operation is called a restore. In order to record a process' complete state, both its volatile and persistent state must be checkpointed. Several libraries exist for checkpointing volatile state. Some of these libraries feature incremental checkpointing, where only the changes since the last checkpoint are recorded in the next checkpoint. Such incremental checkpointing is advantageous since otherwise, the time taken for each successive checkpoint becomes larger and larger. Also, when checkpointing is done in increments, we can restore state to any of the previous checkpoints; a vital feature for adaptive applications. This thesis presents a user-level incremental checkpointing library for files: Metamori. This brings the advantages of incremental memory checkpointing to files as well, thereby providing a low-overhead approach to checkpoint persistent state. Thus, the complete state of an application can now be incrementally checkpointed, as compared to earlier approaches where volatile state was checkpointed incrementally but persistent state had no such facilities. / Master of Science

<p>Fault tolerance is due to the semiconductor technology development important, not only for safety-critical systems but also for general-purpose (non-safety critical) systems. However, instead of guaranteeing that deadlines always are met, it is for general-purpose systems important to minimize the average execution time (AET) while ensuring fault tolerance. For a given job and a soft (transient) no-error probability, we define mathematical formulas for AET using voting (active replication), rollback-recovery with checkpointing (RRC) and a combination of these (CRV) where bus communication overhead is included. And, for a given multi-processor system-on-chip (MPSoC), we define integer linear programming (ILP) models that minimize the AET including bus communication overhead when: (1) selecting the number of checkpoints when using RRC or a combination where RRC is included, (2) finding the number of processors and job-to-processor assignment when using voting or a combination where voting is used, and (3) defining fault tolerance scheme (voting, RRC or CRV) per job and defining its usage for each job. Experiments demonstrate significant savings in AET.</p>

A computação em grid (ou computação em grade) emergiu como uma área de pesquisa importante por permitir o compartilhamento de recursos computacionais geograficamente distribuídos entre vários usuários. Contudo, a heterogeneidade e a dinâmica do comportamento dos recursos em ambientes de grid tornam complexos o desenvolvimento e a execução de aplicações. OurGrid é uma plataforma de software que procura contornar estas dificuldades: além de permitir a execução de aplicações distribuídas em ambientes de computação em grid, oferece e gerencia um esquema de troca de favores entre usuários. Neste esquema, instituições (ou usuários) que possuam recursos ociosos podem oferecê-los a outros que deles necessitem. Quanto mais um domínio oferecer recursos ao grid, mais será favorecido quando precisar, ou seja, terá prioridade mais alta quando requisitar máquinas ao grid. O software MyGrid é o principal componente do OurGrid. É através dele que o usuário interage com o grid, submetendo e gerenciando suas aplicações. No modelo de execução do MyGrid, as tarefas são lançadas por um nó central que coordena todo o escalonamento de tarefas que serão executadas no grid. Este nó apresenta uma fragilidade caracterizada na literatura como "ponto único de falhas", pois seu colapso faz com que os resultados do processamento corrente sejam perdidos. Isto pode significar horas ou, até mesmo, dias de processamento perdido, dependendo das aplicações. Visando suprir esta deficiência, este trabalho descreve o funcionamento e a implementação de um mecanismo de checkpointing (ou salvamento de estado), usado como base para a recuperação por retorno, que permite ao sistema voltar a um estado consistente, minimizando a perda de dados, após uma falha no nó central do MyGrid. Assim, ele salva, de forma estável, o estado da aplicação (estruturas de dados e informações de controle imprescindíveis) capaz de restaurar o sistema após o colapso, oferecendo uma alternativa à sua característica de ponto único de falhas. Os checkpoints são obtidos e salvos a cada mudança de estado do escalonador de tarefas do nó central. A eficiência do mecanismo de recuperação é comprovada através de experimentos que exercitam este mecanismo em cenários com diferentes características, visando validar e avaliar o impacto real no desempenho do MyGrid. / The grid computing has emerged as an important research area because it allows sharing geographically distributed computing resources among several users. However, resources in a grid are highly heterogeneous and dynamic, turning complex the development and the execution of applications. OurGrid is a software platform that intends to reduce these difficulties. Besides allowing the execution of distributed applications in grid environments, it offers and gives support to an exchange of favors between users. In this way, institutions (or users) that have idle resources can offer them to other users. The more resources a domain offers to the grid, the more it will be favored when in need. It will have higher priority when requesting machines to grid. MyGrid software is the main component of OurGrid: it constitutes the interface for user interaction as well as application submission and management. In the execution model of MyGrid, tasks are launched by a central node (home-machine), which manages the scheduling of tasks to be executed in the grid. This node constitutes a "single point of failure", because its crash causes the loss of results of the previous processing. Depending on the particular applications, this loss can be the result of hours or days of processing time. This dissertation aims to reduce the consequences of this problem offering an alternative to the single point of failure: here is proposed and implemented a checkpointing mechanism, used as basis for the rollback recovery. Checkpoints are taken synchronously with the state changes of the scheduler on the central node. After a failure affecting the home-machine of MyGrid, the system recovers information on the state of the application (data structures and essential control information) and results of previous computation, saved in stable storage, minimizing the loss of data. The efficiency of the recovery mechanism and its impact over MyGrid are evaluated through experiments that exercise this mechanism in scenarios with different characteristics.

A computação em grid (ou computação em grade) emergiu como uma área de pesquisa importante por permitir o compartilhamento de recursos computacionais geograficamente distribuídos entre vários usuários. Contudo, a heterogeneidade e a dinâmica do comportamento dos recursos em ambientes de grid tornam complexos o desenvolvimento e a execução de aplicações. OurGrid é uma plataforma de software que procura contornar estas dificuldades: além de permitir a execução de aplicações distribuídas em ambientes de computação em grid, oferece e gerencia um esquema de troca de favores entre usuários. Neste esquema, instituições (ou usuários) que possuam recursos ociosos podem oferecê-los a outros que deles necessitem. Quanto mais um domínio oferecer recursos ao grid, mais será favorecido quando precisar, ou seja, terá prioridade mais alta quando requisitar máquinas ao grid. O software MyGrid é o principal componente do OurGrid. É através dele que o usuário interage com o grid, submetendo e gerenciando suas aplicações. No modelo de execução do MyGrid, as tarefas são lançadas por um nó central que coordena todo o escalonamento de tarefas que serão executadas no grid. Este nó apresenta uma fragilidade caracterizada na literatura como "ponto único de falhas", pois seu colapso faz com que os resultados do processamento corrente sejam perdidos. Isto pode significar horas ou, até mesmo, dias de processamento perdido, dependendo das aplicações. Visando suprir esta deficiência, este trabalho descreve o funcionamento e a implementação de um mecanismo de checkpointing (ou salvamento de estado), usado como base para a recuperação por retorno, que permite ao sistema voltar a um estado consistente, minimizando a perda de dados, após uma falha no nó central do MyGrid. Assim, ele salva, de forma estável, o estado da aplicação (estruturas de dados e informações de controle imprescindíveis) capaz de restaurar o sistema após o colapso, oferecendo uma alternativa à sua característica de ponto único de falhas. Os checkpoints são obtidos e salvos a cada mudança de estado do escalonador de tarefas do nó central. A eficiência do mecanismo de recuperação é comprovada através de experimentos que exercitam este mecanismo em cenários com diferentes características, visando validar e avaliar o impacto real no desempenho do MyGrid. / The grid computing has emerged as an important research area because it allows sharing geographically distributed computing resources among several users. However, resources in a grid are highly heterogeneous and dynamic, turning complex the development and the execution of applications. OurGrid is a software platform that intends to reduce these difficulties. Besides allowing the execution of distributed applications in grid environments, it offers and gives support to an exchange of favors between users. In this way, institutions (or users) that have idle resources can offer them to other users. The more resources a domain offers to the grid, the more it will be favored when in need. It will have higher priority when requesting machines to grid. MyGrid software is the main component of OurGrid: it constitutes the interface for user interaction as well as application submission and management. In the execution model of MyGrid, tasks are launched by a central node (home-machine), which manages the scheduling of tasks to be executed in the grid. This node constitutes a "single point of failure", because its crash causes the loss of results of the previous processing. Depending on the particular applications, this loss can be the result of hours or days of processing time. This dissertation aims to reduce the consequences of this problem offering an alternative to the single point of failure: here is proposed and implemented a checkpointing mechanism, used as basis for the rollback recovery. Checkpoints are taken synchronously with the state changes of the scheduler on the central node. After a failure affecting the home-machine of MyGrid, the system recovers information on the state of the application (data structures and essential control information) and results of previous computation, saved in stable storage, minimizing the loss of data. The efficiency of the recovery mechanism and its impact over MyGrid are evaluated through experiments that exercise this mechanism in scenarios with different characteristics.

Fault tolerance is due to the semiconductor technology development important, not only for safety-critical systems but also for general-purpose (non-safety critical) systems. However, instead of guaranteeing that deadlines always are met, it is for general-purpose systems important to minimize the average execution time (AET) while ensuring fault tolerance. For a given job and a soft (transient) no-error probability, we define mathematical formulas for AET using voting (active replication), rollback-recovery with checkpointing (RRC) and a combination of these (CRV) where bus communication overhead is included. And, for a given multi-processor system-on-chip (MPSoC), we define integer linear programming (ILP) models that minimize the AET including bus communication overhead when: (1) selecting the number of checkpoints when using RRC or a combination where RRC is included, (2) finding the number of processors and job-to-processor assignment when using voting or a combination where voting is used, and (3) defining fault tolerance scheme (voting, RRC or CRV) per job and defining its usage for each job. Experiments demonstrate significant savings in AET.

A computação em grid (ou computação em grade) emergiu como uma área de pesquisa importante por permitir o compartilhamento de recursos computacionais geograficamente distribuídos entre vários usuários. Contudo, a heterogeneidade e a dinâmica do comportamento dos recursos em ambientes de grid tornam complexos o desenvolvimento e a execução de aplicações. OurGrid é uma plataforma de software que procura contornar estas dificuldades: além de permitir a execução de aplicações distribuídas em ambientes de computação em grid, oferece e gerencia um esquema de troca de favores entre usuários. Neste esquema, instituições (ou usuários) que possuam recursos ociosos podem oferecê-los a outros que deles necessitem. Quanto mais um domínio oferecer recursos ao grid, mais será favorecido quando precisar, ou seja, terá prioridade mais alta quando requisitar máquinas ao grid. O software MyGrid é o principal componente do OurGrid. É através dele que o usuário interage com o grid, submetendo e gerenciando suas aplicações. No modelo de execução do MyGrid, as tarefas são lançadas por um nó central que coordena todo o escalonamento de tarefas que serão executadas no grid. Este nó apresenta uma fragilidade caracterizada na literatura como "ponto único de falhas", pois seu colapso faz com que os resultados do processamento corrente sejam perdidos. Isto pode significar horas ou, até mesmo, dias de processamento perdido, dependendo das aplicações. Visando suprir esta deficiência, este trabalho descreve o funcionamento e a implementação de um mecanismo de checkpointing (ou salvamento de estado), usado como base para a recuperação por retorno, que permite ao sistema voltar a um estado consistente, minimizando a perda de dados, após uma falha no nó central do MyGrid. Assim, ele salva, de forma estável, o estado da aplicação (estruturas de dados e informações de controle imprescindíveis) capaz de restaurar o sistema após o colapso, oferecendo uma alternativa à sua característica de ponto único de falhas. Os checkpoints são obtidos e salvos a cada mudança de estado do escalonador de tarefas do nó central. A eficiência do mecanismo de recuperação é comprovada através de experimentos que exercitam este mecanismo em cenários com diferentes características, visando validar e avaliar o impacto real no desempenho do MyGrid. / The grid computing has emerged as an important research area because it allows sharing geographically distributed computing resources among several users. However, resources in a grid are highly heterogeneous and dynamic, turning complex the development and the execution of applications. OurGrid is a software platform that intends to reduce these difficulties. Besides allowing the execution of distributed applications in grid environments, it offers and gives support to an exchange of favors between users. In this way, institutions (or users) that have idle resources can offer them to other users. The more resources a domain offers to the grid, the more it will be favored when in need. It will have higher priority when requesting machines to grid. MyGrid software is the main component of OurGrid: it constitutes the interface for user interaction as well as application submission and management. In the execution model of MyGrid, tasks are launched by a central node (home-machine), which manages the scheduling of tasks to be executed in the grid. This node constitutes a "single point of failure", because its crash causes the loss of results of the previous processing. Depending on the particular applications, this loss can be the result of hours or days of processing time. This dissertation aims to reduce the consequences of this problem offering an alternative to the single point of failure: here is proposed and implemented a checkpointing mechanism, used as basis for the rollback recovery. Checkpoints are taken synchronously with the state changes of the scheduler on the central node. After a failure affecting the home-machine of MyGrid, the system recovers information on the state of the application (data structures and essential control information) and results of previous computation, saved in stable storage, minimizing the loss of data. The efficiency of the recovery mechanism and its impact over MyGrid are evaluated through experiments that exercise this mechanism in scenarios with different characteristics.

This thesis investigates two alternative approaches, referred to as automatic- and semi-automatic replay, which can be used to prevent data loss due to a certain set of unforeseen events at Bolagsverket, the Swedish Companies Registration Office. The approaches make it possible to recover the correct data from a database that belongs to a stateless distributed system and that contains erroneous- or inaccurate information due to past faults. Both approaches utilize log-based rollback-recovery techniques but make different assumptions regarding the deterministic behaviour of Bolagsverket’s systems. A stateless distributed system logs all received messages during failure-free operation. During recovery, automatic replay recovers the data by enabling the system to re-process the logged messages. In contrast, semi-automatic replay recovers data by utilizing the logged messages to enable officials at Bolagsverket to manually redo lost work in a controlled manner. Proof-of-concept implementations of the two replay approaches are developed on a simplified model that resembles one of Bolagsverket’s electronic services, yet that is general to any stateless system that communicates asynchronously using JMS messages and synchronously using XML sent over HTTP. The theoretical- and performance evaluation was conducted with the aim of producing results general to any system with similar characteristics to those of the model. The results suggest that the failure-free overhead at Bolagsverket is approximately 100 milliseconds per logged message, and that around 3 gigabytes of data must be stored in order to recover one average day’s operation. Further, automatic replay successfully manages to recover one average day’s operation in around 70 minutes. Semi-automatic replay is calculated to require, at a maximum, one workday to recover the same amount of data. It is assessed that automatic replay is a suitable solution for Bolagsverket if it is proven that their systems are fully deterministic. In other cases, it is assessed that semi-automatic replay can be utilized. It is however recommended that further evaluations are conducted before the approaches are implemented in a production environment.

Context-dependent applications are relatively complex due to their multiple variations caused by context activation, especially in the presence of unanticipated adaptation. Testing these systems is challenging, as it is hard to reproduce the same execution environments. Therefore, a software failure caused by bugs is no exception. This paper presents a rollback mechanism to recover from software failures as part of a role-based runtime with support for unanticipated adaptation. The mechanism performs checkpoints before each adaptation and employs specialized sensors to detect bugs resulting from recent configuration changes. When the runtime detects a bug, it assumes that the bug belongs to the latest configuration. The runtime rolls back to the recent checkpoint to recover and subsequently notifes the developer to fix the bug and re-applying the adaptation through unanticipated adaptation. We prototype the concept as part of our role-based runtime engine LyRT and demonstrate the applicability of the rollback recovery mechanism for unanticipated adaptation in erroneous situations.

A distributed system is defined as a collection of autonomous computers connected by a network, and with the appropriate distributed software for the system to be seen by users as a single entity capable of providing computing facilities. Distributed systems with centralised control have a distinguished control node, called leader node. The main role of a leader node is to distribute and manage shared resources in a resource-efficient manner. A distributed system with centralised control can use stream processing networks for communication. In a stream processing system, applications typically act as continuous queries, ingesting data continuously, analyzing and correlating the data, and generating a stream of results. Fault tolerance is the ability of a system to process the information, even if it happens any failure or anomaly in the system. Fault tolerance has become an important requirement for distributed systems, due to the possibility of failure has currently risen to the increase in number of nodes and the runtime of applications in distributed system. Therefore, to resolve this problem, it is important to add fault tolerance mechanisms order to provide the internal capacity to preserve the execution of the tasks despite the occurrence of faults. If the leader on a centralised control system fails, it is necessary to elect a new leader. While leader election has received a lot of attention in message-passing systems, very few solutions have been proposed for shared memory systems, as we propose. In addition, rollback-recovery strategies are important fault tolerance mechanisms for distributed systems, since that it is based on storing information into a stable storage in failure-free state and when a failure affects a node, the system uses the information stored to recover the state of the node before the failure appears. In this thesis, we are focused on creating two fault tolerance mechanisms for distributed systems with centralised control that uses stream processing for communication. These two mechanism created are leader election and log-based rollback-recovery, implemented using LPEL. The leader election method proposed is based on an atomic Compare-And-Swap (CAS) instruction, which is directly available on many processors. Our leader election method works with idle nodes, meaning that only the non-busy nodes compete to become the new leader while the busy nodes can continue with their tasks and later update their leader reference. Furthermore, this leader election method has short completion time and low space complexity. The log-based rollback-recovery method proposed for distributed systems with stream processing networks is a novel approach that is free from domino effect and does not generate orphan messages accomplishing the always-no-orphans consistency condition. Additionally, this approach has lower overhead impact into the system compared to other approaches, and it is a mechanism that provides scalability, because it is insensitive to the number of nodes in the system.