No description available.

Scalability is an important problem in epidemiological applications that simulate complex intervention scenarios over large datasets. Indemics is one such interactive data intensive framework for High-performance computing (HPC) based large-scale epidemic simulations. In the Indemics framework, interventions are supplied from an external, standalone database which proved to be an effective way of implementing interventions. Although this setup performs well for simple interventions and small datasets, performance and scalability of complex interventions and large datasets remain an issue. In this thesis, we present IndemicsXC, a scalable and massively parallel high-performance data engine for Indemics in a supercomputing environment. IndemicsXC has the ability to implement complex interventions over large datasets. Our distributed database solution retains the simplicity of Indemics by using the same SQL query interface for expressing interventions. We show that our solution implements the most complex interventions by intelligently offloading them to the supercomputer nodes and processing them in parallel. We present an extensive performance evaluation of our database engine with the help of various intervention case studies over synthetic population datasets. The evaluation of our parallel and distributed database framework illustrates its scalability over standalone database. Our results show that the distributed data engine is efficient as it is parallel, scalable and cost-efficient means of implementing interventions. The proposed cost-model in this thesis could be used to approximate intervention query execution time with decent accuracy. The usefulness of our distributed database framework could be leveraged for fast, accurate and sensible decisions by the public health officials during an outbreak. Finally, we discuss the considerations for using distributed databases for driving large-scale simulations. / Master of Science

Nous proposons dans ces travaux des algorithmes distribués de clusterisation destinés à répondre à la problématique de la croissance des réseaux. Après avoir donné une spécification pour ce problème, nous fournissons un premier algorithme distribué à base de marches aléatoires pour le résoudre. Cet algorithme n’utilise que des informations locales, et utilise des marches aléatoires pour construire en parallèle des ensembles connexes de noeuds appelés les coeurs des clusters, auxquels on ajoute des noeuds adjacents. La taille de chaque coeur est comprise entre 2 et un paramètre de l’algorithme. L’algorithme garantit que si deux clusters sont adjacents, au moins l’un d’entre eux a un coeur de taille maximale. Un deuxième algorithme, adaptatif à la mobilité, garantit en plus de ces propriétés que la reconstruction consécutive à un changement topologique est locale. Cette propriété différencie notre solution des nombreuses solutions existantes : elle permet d’éviter des destructions en chaîne suite à un changement de topologie. Nous présentons enfin un algorithme de clustering auto-stabilisant qui conserve les propriétés des algorithmes précédents en y ajoutant la tolérance aux pannes. Grâce au parallélisme de la construction des clusters et au caractère local des reconstructions de clusters, ces algorithmes passent à l'échelle, ce qui est confirmé par les simulations que nous avons menées. / We propose in this work distributed clustering algorithms designed to address the problem of growing networks. After giving a specification for this problem, we provide a first distributed algorithm based on random walks to solve it. This algorithm uses only local information,and uses random walks to build connected sets of nodes called cores of clusters in parallel, to which we add adjacent nodes. The size of each core is between 2 and a parameter of the algorithm. The algorithm guarantees that if two clusters are adjacent, at least one of them has a core of maximum size. A second, mobility-adaptive, algorithm ensures, besides those properties, that the reconfiguration following a topological change is local. This property differentiates our solution from many solutions : it avoids chain destruction following a topology change. Finally, we present a self-stabilizing clustering algorithm that preserves the properties of previous algorithms and adds fault tolerance. With the parallel construction of clusters and the local nature of the reconstruction of clusters, these algorithms guarantee the scabability, which is confirmed by simulations.

There is a range of tools and techniques in the realm of information security that can be used to enhance the security of a distributed network protocol and some of them introduce new problems. A security analysis of the distributed network protocol SLMP is made and three vulnerabilities are identified; messages can be intercepted and tampered with, nodes can fake id, and leader nodes can do a lot of harm if they are malicious. Three versions of SLMP that aims to remedy these vulnerabilities are implemented and the results show that while they remedy the vulnerabilities some of them introduce new problems.

Nos últimos anos, houve um aumento da complexidade e variedade de serviços disponíveis na Internet, fato que tem levado à busca por técnicas eficientes de roteamento de requisições de um cliente ao melhor servidor disponível, sendo uma delas conhecida como application layer anycast (ALA). O objetivo deste mestrado é elaborar meios eficientes de prover anycast na camada de aplicação com qualidade de serviço no contexto de computação em nuvem. Para atingir esse objetivo, um novo sistema foi proposto (GALA, Global Application Layer Anycast). Ele herda características de um outro sistema existente e emprega a geolocalização como diferencial, a fim de melhorar o desempenho geral do algoritmo. Experimentos foram realizados por meio de simulação e os resultados mostraram que esse novo sistema, comparado ao algoritmo herdado, mantém a eficiência das requisições realizadas pelos clientes e diminui consideravelmente o tempo de latência dessas operações. Ainda, o sistema proposto foi desenvolvido em um ambiente real a fim de fortalecer os resultados das simulações. Com os resultados obtidos, o sistema modelado foi validado e sua eficácia confirmada. / In the past years, the complexity and variety of available services expanded in the Internet, fact that is drawing attention of many researchers that wish to find out efficient techniques of routing client requests to the closest server, being one of them known as application layer anycast (ALA). Thus, the objective of this research is to elaborate ways to offer application layer anycast that are scalable and select the closest servers with the shortest latency possible, in the context of cloud computing. To achieve this goal, a new system was proposed (GALA, Global Application Layer Anycast). It inherits features from an existing system and applies geolocation to improve its overall performance. Simulation results indicated that the new system, compared to its antecessor, has the same efficiency but decreases considerably the requests latency. Yet, the proposed system was deployed in a real environment to strengthen the simulations results. With the obtained data, the modeled system was validated and its efficiency confirmed.

Monitoring data-intensive scientific infrastructures in real-time such as jobs, data transfers, and hardware failures is vital for efficient operation. Due to the high volume and velocity of events that are produced, traditional methods are no longer optimal. Several techniques, as well as enabling architectures, are available to support the Big Data issue. In this respect, this thesis complements existing survey work by contributing an extensive literature review of both traditional and emerging Big Data architecture. Scalability, low-latency, fault-tolerance, and intelligence are key challenges of the traditional architecture. However, Big Data technologies and approaches have become increasingly popular for use cases that demand the use of scalable, data intensive processing (parallel), and fault-tolerance (data replication) and support for low-latency computations. In the context of a scalable data store and analytics platform for monitoring data-intensive scientific infrastructure, Lambda Architecture was adapted and evaluated on the Worldwide LHC Computing Grid, which has been proven effective. This is especially true for computationally and data-intensive use cases. In this thesis, an efficient strategy for the collection and storage of large volumes of data for computation is presented. By moving the transformation logic out from the data pipeline and moving to analytics layers, it simplifies the architecture and overall process. Time utilised is reduced, untampered raw data are kept at storage level for fault-tolerance, and the required transformation can be done when needed. An optimised Lambda Architecture (OLA), which involved modelling an efficient way of joining batch layer and streaming layer with minimum code duplications in order to support scalability, low-latency, and fault-tolerance is presented. A few models were evaluated; pure streaming layer, pure batch layer and the combination of both batch and streaming layers. Experimental results demonstrate that OLA performed better than the traditional architecture as well the Lambda Architecture. The OLA was also enhanced by adding an intelligence layer for predicting data access pattern. The intelligence layer actively adapts and updates the model built by the batch layer, which eliminates the re-training time while providing a high level of accuracy using the Deep Learning technique. The fundamental contribution to knowledge is a scalable, low-latency, fault-tolerant, intelligent, and heterogeneous-based architecture for monitoring a data-intensive scientific infrastructure, that can benefit from Big Data, technologies and approaches.

In this paper, we discuss the consensus problem for synchronous distributed systems with orderly crash failures. For a synchronous distributed system of n processes with up to t crash failures and f failures actually occur, first, we present a bivalency argument proof to solve the open problem of proving the lower bound, min (t + 1, f + 2) rounds, for early-stopping synchronous consensus with orderly crash failures, where t < n - 1. Then, we extend the system model with orderly crash failures to a new model in which a process is allowed to send multiple messages to the same destination process in a round and the failing processes still respect the order specified by the protocol in sending messages. For this new model, we present a uniform consensus protocol, in which all non-faulty processes always decide and stop immediately by the end of f + 1 rounds. We prove that the lower bound of early stopping protocols for both consensus and uniform consensus are f + 1 rounds under the new model, and our proposed protocol is optimal. / Singapore-MIT Alliance (SMA)

The use of service-oriented (SO) distributed systems is increasing. Within service orientation web services (WS) are the de facto standard for implementing service-oriented systems. The consumers of WS want to get uninterrupted and reliable service from the service providers. But WS providers cannot always provide services in the expected level due to faults and failures in the system. As a result the fault management of these systems is becoming crucial. This work presents a distributed event-driven architecture for fault management of Web Services. According to the architecture the managed WS report different events to the event databases. From event databases these events are sent to the event processors. The event processors are distributed over the network. They process the events, detect fault scenarios in the event stream and manage faults in the WS.

We are addressing two aspects of vision based system development that are not fully exploited in current frameworks: abstraction over low-level details and high-level module reusability. Through an evaluation of existing frameworks, we relate these shortcomings to the lack of systematic classification of sub-tasks in vision based system development. Our approach for addressing these two issues is to classify vision into decoupled sub-tasks, hence defining a clear scope for a vision based system development framework and its sub-components. Firstly, we decompose the task of vision system development into data management and processing. We then proceed to further decompose data management into three components: data access, conversion and transportation. To verify our approach for vision system development we present two frameworks: the Vision Utility (VU) framework for providing abstraction over the data management component; and the Hive framework for providing the data transportation and high-level code reuse. VU provides the data management functionality for developers while hiding the low-level system details through a simple yet flexible Application Programming Interface (API). VU mediates the communication between the developer's application, vision processing modules, and data sources by utilizing different frameworks for data access, conversion and transportation (Hive). We demonstrate VU's ability for providing abstraction over low-level system details through the examination of a vision system developed using the framework. Hive is a standalone event based framework for developing distributed vision based systems. Hive provides simple high-level methods for managing communication, control and configuration of reusable components. We verify the requirements of Hive (reusability and abstraction over inter-module data transportation) by presenting a number of different systems developed on the framework using a set of reusable modules. Through this work we aim to demonstrate that this novel approach for vision system development could fundamentally change vision based system development by addressing the necessary abstraction, and promoting high-level code reuse.

A <i>web crawler</i> is a program that "walks" the Web to gather web resources. In order to scale to the ever-increasing Web, multiple crawling agents may be deployed in a distributed fashion to retrieve web data co-operatively. A common approach is to divide the Web into many partitions with an agent assigned to crawl within each one. If an agent obtains a web resource that is not from its partition, the resource will be transferred to the rightful owner. This thesis proposes a novel approach to distributed web data gathering by partitioning the Web into topics. The proposed approach employs multiple focused crawlers to retrieve pages from various topics. When a crawler retrieves a page of another topic, it transfers the page to the appropriate crawler. This approach is known as <i>topic-oriented collaborative web crawling</i>. An implementation of the system was built and experimentally evaluated. In order to identify the topic of a web page, a topic classifier was incorporated into the crawling system. As the classifier categorizes only English pages, a language identifier was also introduced to distinguish English pages from non-English ones. From the experimental results, we found that redundance retrieval was low and that a resource, retrieved by an agent, is six times more likely to be retained than a system that uses conventional hashing approach. These numbers were viewed as strong indications that <i>topic-oriented collaborative web crawling system</i> is a viable approach to web data gathering.