No description available.

In recent years, adaptive HTTP streaming protocols have become the de facto standard in the industry for the distribution of live and video-on-demand content over the Internet. In this thesis, we solve the problem of distributing adaptive HTTP live video streams to a large number of viewers using peer-to-peer (P2P) overlays. We do so by assuming that our solution must deliver a level of quality of user experience which is the same as a CDN while trying to minimize the load on the content provider’s infrastructure. Besides that, in the design of our solution, we take into consideration the realities of the HTTP streaming protocols, such as the pull-based approach and adaptive bitrate switching. The result of this work is a system which we call SmoothCache that provides CDN-quality adaptive HTTP live streaming utilizing P2P algorithms. Our experiments on a real network of thousands of consumer machines show that, besides meeting the the CDN-quality constraints, SmoothCache is able to consistently deliver up to 96% savings towards the source of the stream in a single bitrate scenario and 94% in a multi-bitrate scenario. In addition, we have conducted a number of pilot deployments in the setting of large enterprises with the same system, albeit tailored to private networks. Results with thousands of real viewers show that our platform provides an average offloading of bottlenecks in the private network of 91.5%. These achievements were made possible by advancements in multiple research areas that are also presented in this thesis. Each one of the contributions is novel with respect to the state of the art and can be applied outside of the context of our application. However, in our system they serve the purposes described below. We built a component-based event-driven framework to facilitate the development of our live streaming application. The framework allows for running the same code both in simulation and in real deployment. In order to obtain scalability of simulations and accuracy, we designed a novel flow-based bandwidth emulation model. In order to deploy our application on real networks, we have developed a network library which has the novel feature of providing on-the-fly prioritization of transfers. The library is layered over the UDP protocol and supports NAT Traversal techniques. As part of this thesis, we have also improved on the state of the art of NAT Traversal techniques resulting in higher probability of direct connectivity between peers on the Internet. Because of the presence of NATs on the Internet, discovery of new peers and collection of statistics on the overlay through peer sampling is problematic. Therefore, we created a peer sampling service which is NAT-aware and provides one order of magnitude fresher samples than existing peer sampling protocols. Finally, we designed SmoothCache as a peer-assisted live streaming system based on a distributed caching abstraction. In SmoothCache, peers retrieve video fragments from the P2P overlay as quickly as possible or fall back to the source of the stream to keep the timeliness of the delivery. In order to produce savings, the caching system strives to fill up the local cache of the peers ahead of playback by prefetching content. Fragments are efficiently distributed by a self-organizing overlay network that takes into account many factors such as upload bandwidth capacity, connectivity constraints, performance history and the currently being watched bitrate. / <p>QC 20131122</p>

A growing use for serverless computing is large parallel data processing applications that take advantage of its on-demand scalability. Because individual serverless compute nodes, which are called functions, run in isolated containers, a major challenge with this paradigm is transferring temporary computation data between functions. Previous works have performed inter-function communication using object storage, which is slow, or in-memory databases, which are expensive. We evaluate the use of direct network connections between functions to overcome these limitations. Although function containers block incoming connections, we are able to bypass this restriction using standard NAT traversal techniques. By using an external server, we implement TCP hole punching to establish direct TCP connections between functions. In addition, we develop a communications framework to manage NAT traversal and data flow for applications using direct network connections. We evaluate this framework with a reduce-by-key application compared to an equivalent version that uses object storage for communication. For a job with 100+ functions, our TCP implementation runs 4.7 times faster at almost half the cost. / Master of Science / Serverless computing is a branch of cloud computing where users can remotely run small programs, called "functions," and pay only based on how long they run. A growing use for serverless computing is running large data processing applications that use many of these serverless functions at once, taking advantage of the fact that serverless programs can be started quickly and on-demand. Because serverless functions run on isolated networks from each other and can only make outbound connections to the public internet, a major challenge with this paradigm is transferring temporary computation data between functions. Previous works have used separate types of cloud storage services in combination with serverless computing to allow functions to exchange data. However, hard-drive--based storage is slow and memory-based storage is expensive. We evaluate the use of direct network connections between functions to overcome these limitations. Although functions cannot receive incoming network connections, we are able to bypass this restriction by using a standard networking technique called Network Address Translation (NAT) traversal. We use an external server as an initial relay to setup a network connection between two functions such that once the connection is established, the functions can communicate directly with each other without using the server anymore. In addition, we develop a communications framework to manage NAT traversal and data flow for applications using direct network connections. We evaluate this framework with an application for combining matching data entries and compare it to an equivalent version that uses storage based on hard drives for communication. For a job with over 100 functions, our implementation using direct network connections runs 4.7 times faster at almost half the cost.

Um dos grandes avanços na área da computação nas últimas décadas refere-se aos sistemas distribuídos. O uso de tais sistemas tem crescido vertiginosamente nos últimos anos. Com o surgimento e uso de tecnologias de redes sem-fio, os sistemas ganharam mobilidade e dinamismo. Pessoas podem entrar ou sair de seus locais de trabalho, de entretenimento ou mesmo de suas casas com seus computadores móveis; redes podem ser formadas ou extintas conforme a dinâmica de um grupo de pessoas; e novos serviços podem ser disponibilizados ou interrompidos. Em sistemas como tais, um sistema de localização de serviços automático é fundamental para acompanhar as contínuas modificações na rede. A localização de serviços permite que, em uma rede de endereçamento estático ou dinâmico, dispositivos e serviços possam ser encontrados de modo completamente automático. A presente dissertação propõe um sistema de localização lógica (endereçamento) de serviços e dispositivos em uma rede local ou geograficamente distribuída, indo ao encontro às necessidades de controle de sistemas distribuídos dinâmicos, contribuindo ao mesmo tempo com questões de usabilidade e configuração automática de redes. O sistema, também, busca resolver questões relacionadas ao uso dos serviços entre redes protegidas por firewalls ou que utilizam configurações de NAT. / In the last decades, one of the most important progresses in computation is related to the distributed systems. The usage of these systems has increased a lot, in the last years. Due the emergence of wireless technologies, systems have acquired mobility and dynamism. People can enter or leave places, such as home, workplaces and entertainment spots, caring their mobile computers; networks can be formed or extinguished in accordance with the people dynamism; and new services can be available or interrupted. At these systems, a service location system is needed to keep track of the unceasing modifications in the network. The service location enables that services and devices can be found, at dynamic or static networks. This master thesis proposes a system for location of the logical addresses of services and devices at local or wide area networks. This work contributes to issues related to distributed systems control, usability and automatic network configuration. This system also solves questions related to service usage between networks using firewalls or NAT. Besides that, other points related to network interconnections in distributed systems, for example, using the Internet, are raised and solved.

Um dos grandes avanços na área da computação nas últimas décadas refere-se aos sistemas distribuídos. O uso de tais sistemas tem crescido vertiginosamente nos últimos anos. Com o surgimento e uso de tecnologias de redes sem-fio, os sistemas ganharam mobilidade e dinamismo. Pessoas podem entrar ou sair de seus locais de trabalho, de entretenimento ou mesmo de suas casas com seus computadores móveis; redes podem ser formadas ou extintas conforme a dinâmica de um grupo de pessoas; e novos serviços podem ser disponibilizados ou interrompidos. Em sistemas como tais, um sistema de localização de serviços automático é fundamental para acompanhar as contínuas modificações na rede. A localização de serviços permite que, em uma rede de endereçamento estático ou dinâmico, dispositivos e serviços possam ser encontrados de modo completamente automático. A presente dissertação propõe um sistema de localização lógica (endereçamento) de serviços e dispositivos em uma rede local ou geograficamente distribuída, indo ao encontro às necessidades de controle de sistemas distribuídos dinâmicos, contribuindo ao mesmo tempo com questões de usabilidade e configuração automática de redes. O sistema, também, busca resolver questões relacionadas ao uso dos serviços entre redes protegidas por firewalls ou que utilizam configurações de NAT. / In the last decades, one of the most important progresses in computation is related to the distributed systems. The usage of these systems has increased a lot, in the last years. Due the emergence of wireless technologies, systems have acquired mobility and dynamism. People can enter or leave places, such as home, workplaces and entertainment spots, caring their mobile computers; networks can be formed or extinguished in accordance with the people dynamism; and new services can be available or interrupted. At these systems, a service location system is needed to keep track of the unceasing modifications in the network. The service location enables that services and devices can be found, at dynamic or static networks. This master thesis proposes a system for location of the logical addresses of services and devices at local or wide area networks. This work contributes to issues related to distributed systems control, usability and automatic network configuration. This system also solves questions related to service usage between networks using firewalls or NAT. Besides that, other points related to network interconnections in distributed systems, for example, using the Internet, are raised and solved.

In this thesis, we explore the use of a centrally-coordinated peer-to-peer overlay as a possible solution to the live streaming problem. Our contribution lies in showing that such approach is indeed feasible given that a number of key challenges are met. The motivation behind exploring an alternative design is that, although a number of approaches have been investigated in the past, e.g. mesh-pull and tree-push, hybrids and best-of-both-worlds mesh-push, no consensus has been reached on the best solution for the problem of peer-to-peer live streaming, despite current deployments and reported successes. In the proposed system, we model sender/receiver peer assignments as an optimization problem. Optimized peer selection based on multiple utility factors, such as bandwidth availability, delays and connectivity compatibility, make it possible to achieve large source bandwidth savings and provide high quality of user experience. Clear benefits of our approach are observed when Network Address Translation constraints are present on the network. We have addressed key scalability issues of our platform by parallelizing the heuristic which is the core of our optimization engine and by implementing the resulting algorithm on commodity Graphic Processing Units (GPUs). The outcome is a Linear Sum Assignment Problem (LSAP) solver for time-constrained systems which produces near-optimal results and can be used for any instance of LSAP, i.e. not only in our system.   As part of this work, we also present our experience in working with Network Address Translators (NATs) traversal in peer-to-peer systems. Our contribution in this context is threefold. First, we provide a semi-formal model of state of the art NAT behaviors. Second, we use our model to show which NAT combinations can be theoretically traversed and which not. Last, for each of the combinations, we state which traversal technique should be used. Our findings are confirmed by experimental results on a real network. Finally, we address the problem of reproducibility in testing, debugging and evaluation of our peer-to-peer application. We achieve this by providing a software framework which can be transparently integrated with any already-existing software and which is able to handle concurrency, system time and network events in a reproducible manner. / QC 20110426