Today, distributed applications run at data centre and Internet scales, from intensive data analysis, such as MapReduce; to the dynamic demands of a worldwide audience, such as YouTube. The network is essential to these applications at both scales. To provide adequate support, we must understand the full requirements of the applications, which are revealed by the workloads. In this thesis, we study distributed system applications at different scales to enrich this understanding. Large-scale Internet applications have been studied for years, such as social networking service (SNS), video on demand (VoD), and content delivery networks (CDN). An emerging type of video broadcasting on the Internet featuring crowdsourced live video streaming has garnered attention allowing platforms such as Twitch to attract over 1 million concurrent users globally. To better understand Twitch, we collected real-time popularity data combined with metadata about the contents and found the broadcasters rather than the content drives its popularity. Unlike YouTube and Netflix where content can be cached, video streaming on Twitch is generated instantly and needs to be delivered to users immediately to enable real-time interaction. Thus, we performed a large-scale measurement of Twitchs content location revealing the global footprint of its infrastructure as well as discovering the dynamic stream hosting and client redirection strategies that helped Twitch serve millions of users at scale. We next consider applications that run inside the data centre. Distributed computing applications heavily rely on the network due to data transmission needs and the scheduling of resources and tasks. One successful application, called Hadoop, has been widely deployed for Big Data processing. However, little work has been devoted to understanding its network. We found the Hadoop behaviour is limited by hardware resources and processing jobs presented. Thus, after characterising the Hadoop traffic on our testbed with a set of benchmark jobs, we built a simulator to reproduce Hadoops job traffic With the simulator, users can investigate the connections between Hadoop traffic and network performance without additional hardware cost. Different network components can be added to investigate the performance, such as network topologies, queue policies, and transport layer protocols. In this thesis, we extended the knowledge of networking by investigated two widelyused applications in the data centre and at Internet scale. We (i) studied the most popular live video streaming platform Twitch as a new type of Internet-scale distributed application revealing that broadcaster factors drive the popularity of such platform, and we (ii) discovered the footprint of Twitch streaming infrastructure and the dynamic stream hosting and client redirection strategies to provide an in-depth example of video streaming delivery occurring at the Internet scale, also we (iii) investigated the traffic generated by a distributed application by characterising the traffic of Hadoop under various parameters, (iv) with such knowledge, we built a simulation tool so users can efficiently investigate the performance of different network components under distributed application.

Este trabalho propõe um modelo hierárquico para coleta de dados de monitoramento em sistemas distribuídos. Seu objetivo é proporcionar a análise on-line do comportamento de sistemas e programas distribuídos. O meio escolhido para realizar essa análise foi a visualização. Inicialmente é apresentada uma contextualização sobre monitoramento de sistemas distribuídos. Também são abordados aspectos específicos ao monitoramento de Grid. Após, é analisado um conjunto de ferramentas de monitoramento. Então tem-se a apresentação do modelo proposto. Esse é composto por coletores locais, por uma hierarquia de agregadores e por clientes. É utilizado o modelo push de transmissão de dados e há um mecanismo de subscrição aos coletores. Foi implementado um protótipo do modelo de coleta proposto, que foi utilizado na implementação de um protótipo de ferramenta de monitoramento on-line. Nessa, os dados coletados são fornecidos ao DIMVisual, que é um modelo de integração de dados para visualização. Para visualização, o protótipo utiliza a ferramenta TRIVA, que recebe os dados integrados como entrada. Essa ferramenta foi modificada para gerar uma visualização que é atualizada de maneira on-line. Também foram realizados experimentos para avaliar o tempo necessário para enviar mensagens com diferentes hierarquias e configurações dos coletores. Além disso, foi avaliada a capacidade de o cliente implementado processar os dados recebidos, gerando sua visualização. / This work proposes a hierarchical model for collecting monitoring data from distributed systems. Its goal is to allow the on-line analysis of the behavior of distributed systems and applications. The means we chose to perform this analysis is to generate a visualization of the collected information. In the beginning of this dissertation we present an overview of the monitoring of distributed systems. Aspects that are specific to the monitoring of Grid systems are also reviewed. Next, we have an analysis of a set of monitoring tools. Then we present the proposed model, which is composed by local collectors, an hierarchical structure of aggregators and clients. A push data transmission model is used in the model and it also has a subscription mechanism. A prototype monitoring tool was implemented, integrating the data collection model with DIMVisual and TRIVA. The former is a data integration model whose output is formatted to be used as input for a visualization tool. The later is a visualization tool which, in the prototype, receives the integrated data from DIMVisual. TRIVA generates a visualization of the received information, which is updated in an on-line fashion. In order to evaluate the model, we performed a set of experiments using the prototype. One of the experiments measured the time spent to send data though different hierarchies. In these tests we have also varied the quantity and the configuration of the collectors. In another experiment we evaluated the capacity of the client to process the received data.

A common characteristic among parallel/distributed programming languages is that the one language is used to specify not only the overall organisation of the distributed application, but also the functionality of the application. That is, the connectivity and functionality of processes are specified within a single program. Connectivity and functionality are independent aspects of a distributed application. This thesis shows that these two aspects can be specified separately, therefore allowing application designers to freely concentrate on either aspect in a modular fashion. Two new programming languages have been developed for specifying each aspect. These languages are for loosely coupled distributed applications based on message passing, and have been designed to simplify distributed programming by completely removing all low level interprocess communication. A suite of languages and tools has been designed and developed. It includes the two new languages, parsers, a compilation system to generate intermediate C code that is compiled to binary object modules, a run-time system to create, manage and terminate several distributed applications, and a shell to communicate with the run-tune system. DAL (Distributed Application Language) and DAPL (Distributed Application Process Language) are the new programming languages for the specification and development of process oriented, asynchronous message passing, distributed applications. These two languages have been designed and developed as part of this doctorate in order to specify such distributed applications that execute on a cluster of computers. Both languages are used to specify orthogonal components of an application, on the one hand the organisation of processes that constitute an application, and on the other the interface and functionality of each process. Consequently, these components can be created in a modular fashion, individually and concurrently. The DAL language is used to specify not only the connectivity of all processes within an application, but also a cluster of computers for which the application executes. Furthermore, sub-clusters can be specified for individual processes of an application to constrain a process to a particular group of computers. The second language, DAPL, is used to specify the interface, functionality and data structures of application processes. In addition to these languages, a DAL parser, a DAPL parser, and a compilation system have been designed and developed (in this project). This compilation system takes DAL and DAPL programs to generate object modules based on machine code, one module for each application process. These object modules are used by the Distributed Application System (DAS) to instantiate and manage distributed applications. The DAS system is another new component of this project. The purpose of the DAS system is to create, manage, and terminate many distributed applications of similar and different configurations. The creation procedure incorporates the automatic allocation of processes to remote machines. Application management includes several operations such as deletion, addition, replacement, and movement of processes, and also detection and reaction to faults such as a processor crash. A DAS operator communicates with the DAS system via a textual shell called DASH (Distributed Application SHell). This suite of languages and tools allowed distributed applications of varying connectivity and functionality to be specified quickly and simply at a high level of abstraction. DAL and DAPL programs of several processes may require a few dozen lines to specify as compared to several hundred lines of equivalent C code that is generated by the compilation system. Furthermore, the DAL and DAPL compilation system is successful at generating binary object modules, and the DAS system succeeds in instantiating and managing several distributed applications on a cluster.

Researchers in software engineering proposed design method for distributed applications to construct a set of communicating system components from a global behavior. The joint behaviors of these components must precisely satisfy the specified global behavior. The next concern is to transform the constructed models of these components into executable business processes by ensuring the exchange of asynchronous messages among the generated business processes. The introduction of Service-Oriented Architecture (SOA) has helped to achieve this goal. SOA provides high flexibility in composing loosely-integrated services that can be used among business domains to carry out business transactions; this composition is known as service orchestration. Moreover, SOA supports Model Driven Architecture (MDA) such that services modeled as UML Activity Diagrams (AD) can be transformed into a set of Business Execution Language (BPEL) processes. Many researchers discussed the transformation of UML AD and the Business Process Modeling Notation (BPMN) into BPEL. However, they did not discuss the practical limitations that some of these transformations impose. This thesis addresses the imitations of the transformation from UML AD to BPEL processes using the IBM Rational Software Architect (RSA). We showed here that the tool is unable to create the correct BPEL artifacts from UML AD components in certain cases, for instance when the behavior includes the alternative for receiving single or concurrent messages, a weak loop, or certain choice activities. Furthermore, we provided novel solutions to the transformations in these cases in order to facilitate the transformation from UML AD to BPEL.

Researchers in software engineering proposed design method for distributed applications to construct a set of communicating system components from a global behavior. The joint behaviors of these components must precisely satisfy the specified global behavior. The next concern is to transform the constructed models of these components into executable business processes by ensuring the exchange of asynchronous messages among the generated business processes. The introduction of Service-Oriented Architecture (SOA) has helped to achieve this goal. SOA provides high flexibility in composing loosely-integrated services that can be used among business domains to carry out business transactions; this composition is known as service orchestration. Moreover, SOA supports Model Driven Architecture (MDA) such that services modeled as UML Activity Diagrams (AD) can be transformed into a set of Business Execution Language (BPEL) processes. Many researchers discussed the transformation of UML AD and the Business Process Modeling Notation (BPMN) into BPEL. However, they did not discuss the practical limitations that some of these transformations impose. This thesis addresses the imitations of the transformation from UML AD to BPEL processes using the IBM Rational Software Architect (RSA). We showed here that the tool is unable to create the correct BPEL artifacts from UML AD components in certain cases, for instance when the behavior includes the alternative for receiving single or concurrent messages, a weak loop, or certain choice activities. Furthermore, we provided novel solutions to the transformations in these cases in order to facilitate the transformation from UML AD to BPEL.

<p>The Swedish field of digital X-ray imaging has since several years relied heavily on distributed information systems and digital storage containers. </p><p>To ensure accurate and safe radiological reporting, Swedish software-firm eCare AB delivers a system called Feedback—the first and only quality assurance IT support product of its kind. This thesis covers several aspects of the design and implementation of future versions of this software platform. </p><p>The focus lies on distributed directory services and models for secure and robust data propagation in TCP/IP networks. For data propagation, a new application, InfoBroker, has been designed and implemented to facilitate integration between Feedback and other medical IT support systems. The directory services, introduced in this thesis as the Feedback Directory Services, have been designed on the architectural level. A combination of CORBA and Java Enterprise Edition is suggested as the implementation platform.</p>

Researchers in software engineering proposed design method for distributed applications to construct a set of communicating system components from a global behavior. The joint behaviors of these components must precisely satisfy the specified global behavior. The next concern is to transform the constructed models of these components into executable business processes by ensuring the exchange of asynchronous messages among the generated business processes. The introduction of Service-Oriented Architecture (SOA) has helped to achieve this goal. SOA provides high flexibility in composing loosely-integrated services that can be used among business domains to carry out business transactions; this composition is known as service orchestration. Moreover, SOA supports Model Driven Architecture (MDA) such that services modeled as UML Activity Diagrams (AD) can be transformed into a set of Business Execution Language (BPEL) processes. Many researchers discussed the transformation of UML AD and the Business Process Modeling Notation (BPMN) into BPEL. However, they did not discuss the practical limitations that some of these transformations impose. This thesis addresses the imitations of the transformation from UML AD to BPEL processes using the IBM Rational Software Architect (RSA). We showed here that the tool is unable to create the correct BPEL artifacts from UML AD components in certain cases, for instance when the behavior includes the alternative for receiving single or concurrent messages, a weak loop, or certain choice activities. Furthermore, we provided novel solutions to the transformations in these cases in order to facilitate the transformation from UML AD to BPEL.

The architecture of certain class of services, such as distributed computing, distributed storages or content delivering networks shifts from the traditional client-server model to more scalable and robust peer to peer networks. Providing proper protection to such complex, open and anonymous systems is very complicated. Malicious peers can cooperate and develop sophisticated strategies to bypass existing security mechanisms. Recently, many trust management systems for P2P networks have been proposed. However, their effectiveness is usually tested only against simple malicious strategies. Moreover, a complex comparison of resistance of a particular method is missing. In this thesis, we (1) propose a new trust management system called BubbleTrust and (2) develop a simulation framework for testing trust management systems against various malicious strategies. Our simulation framework defines several criteria which determine the success of each malicious strategy in the network with a given system. We present results of four trust management systems that represent main contemporary approaches and BubbleTrust.

Este trabalho propõe um modelo hierárquico para coleta de dados de monitoramento em sistemas distribuídos. Seu objetivo é proporcionar a análise on-line do comportamento de sistemas e programas distribuídos. O meio escolhido para realizar essa análise foi a visualização. Inicialmente é apresentada uma contextualização sobre monitoramento de sistemas distribuídos. Também são abordados aspectos específicos ao monitoramento de Grid. Após, é analisado um conjunto de ferramentas de monitoramento. Então tem-se a apresentação do modelo proposto. Esse é composto por coletores locais, por uma hierarquia de agregadores e por clientes. É utilizado o modelo push de transmissão de dados e há um mecanismo de subscrição aos coletores. Foi implementado um protótipo do modelo de coleta proposto, que foi utilizado na implementação de um protótipo de ferramenta de monitoramento on-line. Nessa, os dados coletados são fornecidos ao DIMVisual, que é um modelo de integração de dados para visualização. Para visualização, o protótipo utiliza a ferramenta TRIVA, que recebe os dados integrados como entrada. Essa ferramenta foi modificada para gerar uma visualização que é atualizada de maneira on-line. Também foram realizados experimentos para avaliar o tempo necessário para enviar mensagens com diferentes hierarquias e configurações dos coletores. Além disso, foi avaliada a capacidade de o cliente implementado processar os dados recebidos, gerando sua visualização. / This work proposes a hierarchical model for collecting monitoring data from distributed systems. Its goal is to allow the on-line analysis of the behavior of distributed systems and applications. The means we chose to perform this analysis is to generate a visualization of the collected information. In the beginning of this dissertation we present an overview of the monitoring of distributed systems. Aspects that are specific to the monitoring of Grid systems are also reviewed. Next, we have an analysis of a set of monitoring tools. Then we present the proposed model, which is composed by local collectors, an hierarchical structure of aggregators and clients. A push data transmission model is used in the model and it also has a subscription mechanism. A prototype monitoring tool was implemented, integrating the data collection model with DIMVisual and TRIVA. The former is a data integration model whose output is formatted to be used as input for a visualization tool. The later is a visualization tool which, in the prototype, receives the integrated data from DIMVisual. TRIVA generates a visualization of the received information, which is updated in an on-line fashion. In order to evaluate the model, we performed a set of experiments using the prototype. One of the experiments measured the time spent to send data though different hierarchies. In these tests we have also varied the quantity and the configuration of the collectors. In another experiment we evaluated the capacity of the client to process the received data.

Este trabalho propõe um modelo hierárquico para coleta de dados de monitoramento em sistemas distribuídos. Seu objetivo é proporcionar a análise on-line do comportamento de sistemas e programas distribuídos. O meio escolhido para realizar essa análise foi a visualização. Inicialmente é apresentada uma contextualização sobre monitoramento de sistemas distribuídos. Também são abordados aspectos específicos ao monitoramento de Grid. Após, é analisado um conjunto de ferramentas de monitoramento. Então tem-se a apresentação do modelo proposto. Esse é composto por coletores locais, por uma hierarquia de agregadores e por clientes. É utilizado o modelo push de transmissão de dados e há um mecanismo de subscrição aos coletores. Foi implementado um protótipo do modelo de coleta proposto, que foi utilizado na implementação de um protótipo de ferramenta de monitoramento on-line. Nessa, os dados coletados são fornecidos ao DIMVisual, que é um modelo de integração de dados para visualização. Para visualização, o protótipo utiliza a ferramenta TRIVA, que recebe os dados integrados como entrada. Essa ferramenta foi modificada para gerar uma visualização que é atualizada de maneira on-line. Também foram realizados experimentos para avaliar o tempo necessário para enviar mensagens com diferentes hierarquias e configurações dos coletores. Além disso, foi avaliada a capacidade de o cliente implementado processar os dados recebidos, gerando sua visualização. / This work proposes a hierarchical model for collecting monitoring data from distributed systems. Its goal is to allow the on-line analysis of the behavior of distributed systems and applications. The means we chose to perform this analysis is to generate a visualization of the collected information. In the beginning of this dissertation we present an overview of the monitoring of distributed systems. Aspects that are specific to the monitoring of Grid systems are also reviewed. Next, we have an analysis of a set of monitoring tools. Then we present the proposed model, which is composed by local collectors, an hierarchical structure of aggregators and clients. A push data transmission model is used in the model and it also has a subscription mechanism. A prototype monitoring tool was implemented, integrating the data collection model with DIMVisual and TRIVA. The former is a data integration model whose output is formatted to be used as input for a visualization tool. The later is a visualization tool which, in the prototype, receives the integrated data from DIMVisual. TRIVA generates a visualization of the received information, which is updated in an on-line fashion. In order to evaluate the model, we performed a set of experiments using the prototype. One of the experiments measured the time spent to send data though different hierarchies. In these tests we have also varied the quantity and the configuration of the collectors. In another experiment we evaluated the capacity of the client to process the received data.