Processing real-world data requires the ability to analyze data in real-time. Data processing engines like Hadoop come short when results are needed on the fly. Apache Spark's streaming library is increasingly becoming a popular choice as it can stream and analyze a significant amount of data. To showcase and assess the ability of Spark various metrics were designed and operated using data collected from the USGODAE data catalog. The latency of streaming in Apache Spark was measured and analyzed against many nodes in the cluster. Scalability was monitored by adding and removing nodes in the middle of a streaming job. Fault tolerance was verified by stopping nodes in the middle of a job and making sure that the job was rescheduled and completed on other node/s. A full stack application was designed that would automate data collection, data processing and visualizing the results. Google Maps API was used to visualize results by color coding the world map with values from various analytics.

This thesis work addresses the limitations of a web application called the Public Information Officer Monitoring Application (PMA). This application helps Public Information Officers (PIOs) to gather, monitor, sort, store, and report social media data during a crisis event. Before this work, PMA was unable to handle large data sets and as a result, it had not been adequately tested with potential users of the application. This thesis describes changes made to PMA to improve its ability to handle large data sets. After these changes were made, the application was then tested with target users. All test participants found the application useful and relevant to their work. Testing also revealed many ways to improve the usefulness of the application, which were subsequently implemented. The thesis concludes with suggestions for future work and distribution of PMA.

Packet switches are used in the Internet to forward information between a sender and receiver and are the critical bottleneck in the Internet. Without faster packet switch designs, the Internet cannot continue to scale-up to higher data rates. Packet switches must be able to achieve high throughput and low delay. In addition, they must be stable for all traffic loads, must efficiently support variable length packets, and must be scalable to higher link data rates and greater numbers of ports. This dissertation investigates a new combined input and crossbar queued (CICQ) switch architecture. Some unbalanced traffic loads result in instability for input queued (IQ) and CICQ switches. This instability region was modeled, and the cause of the instability was found to be a lack of work conservation at one port. A new burst stabilization protocol was investigated that was shown to stabilize both IQ and CICQ switches. As an added benefit, this new protocol did not require a costly internal switch speed-up. Switching variable length packets in IQ switches requires the segmentation of packets into cells. The process also requires an internal switch speed-up which can be costly. A new method of cell-merging in IQ switches reduced this speed-up. To improve fairness for CICQ switches, a block and transfer method was proposed and evaluated. Implementation feasibility of the CICQ switch was also investigated via a field programmable gate array (FPGA) implementation of key components. Two new designs for round robin arbiters were developed and evaluated. The first of these, a proposed priority-encoder-based round robin arbiter that uses feedback masking, has a lower delay than any known design for an FPGA implementation. The second, an overlapped round robin arbiter design that fully overlaps round robin polling and scheduling, was proposed and shown to be scalable, work conserving, and fair. To allow for multi-cabinet implementation and minimization of the size of the cross point buffers, a distributed input port queue scheduler was investigated. This new scheduler minimizes the amount of buffering needed within the crossbar. The two primary contributions of this dissertation are 1) a complete understanding of the performance characteristics of the CICQ switch, and 2) new methods for improving the performance, stability,and scalability of the CICQ switch. This work has shown that the CICQ switch can be the switch architecture of the future.

Le passage à l’échelle des protocoles de routage est un des problèmes les plus critiques pour les réseauxmobiles sans fil à grande envergure. Dans ce cadre, le concept de clusterisation peut être mis à profit dans lafonction de routage afin d’améliorer les performances de ces réseaux. En premier lieu, cette thèse présentenotre algorithme de clusterisation à 1-saut alpha-SSCA (α-Stability Structure Clustering Algorithm) qui apour objectif d’améliorer la stabilité de la structure des clusters. Un algorithme générique de clusterisationà K-sauts est également proposé en ayant le même but de stabilité visé par alpha-SSCA tout en réduisant lenombre de clusters générés et en étant indépendant de la métrique d’élection des cluster-heads. Ensuite, nousprésentons notre proposition d’un protocole de routage à état des liens des clusters qui exploite les apports denotre mécanisme de clusterisation α-SSCA. Ce protocole, appelé CLSR (Cluster-based Link State Routing),vise à réduire le trafic de contrôle afin d’améliorer les performances du réseau à large échelle. Nous avonsproposé aussi une version hiérarchique du protocole CLSR. Ce protocole de routage introduit une hiérarchiedans la structure des clusters qui permet de réduire le nombre de clusters en groupement des clusters prochesdans un même cluster. L’objectif principal de ce protocole hiérarchique est d’améliorer la scalabilité de CLSRquand le nombre de noeuds dans le réseau augmente considérablement. / Scalability is one of critical challenges for routing protocols in large scale mobile wireless networks. In thiscontext, clustering technique seems a promising approach to overcome the scalability problem. First, we proposea one hop clustering algorithm, alpha-SSCA (alpha-Stability Structure Clustering Algorithm), whichaims to improve the stability of the clusters structure. Second, we present a proposal of a generic K-hopsclustering algorithm which is independent of the metrics used to elect cluster-heads. The main contributionof this last algorithm is to enhance the stability of the clusters structure while reducing the number of clusters.Clustering mechanism is introduced in our proposed routing protocol CLSR (Cluster-based Link StateRouting) in order to reduce the control overhead. The main objective of CLSR is to take profit of the stablestructure of clusters to enhance the network scalability.We propose also a second proactive link-state protocolwhich is based on hierarchical clustering. This protocol makes use of hierarchical clustering to more reducethe routing overhead.

<p>The thesis work is conducted in the division of computer engineering at thedepartment of electrical engineering in Linköping University. During the thesiswork, a configurable Direct Memory Access (DMA) controller was designed andimplemented. The DMA controller runs at 200MHz under 65nm digital CMOS technology. The estimated gate count is 26595.</p><p>The DMA controller has two address generators and can provide two clocksources. It can thus handle data read and write simultaneously. There are 16channels built in the DMA controller, the data width can be 16-bit, 32-bit and64-bit. The DMA controller supports 2D data access by configuring its intelligentlinking table. The DMA is designed for advanced DSP applications and it is notdedicated for cache which has a fixed priority.</p>

The shared memory systems should support parallelization at the computation (multi-core), communication (Network-on-Chip, NoC) and memory architecture levels to exploit the potential performance benefits. These parallel systems supporting shared memory abstraction both in the general purpose and application specific domains are confronting the critical issue of memory consistency. The memory consistency issue arises due to the unconstrained memory operations which leads to the unexpected behavior of shared memory systems. The memory consistency models enforce ordering constraints on the memory operations for the expected behavior of the shared memory systems. The intuitive Sequential Consistency (SC) model enforces strict ordering constraints on the memory operations and does not take advantage of the system optimizations both in the hardware and software. Alternatively, the relaxed memory consistency models relax the ordering constraints on the memory operations and exploit these optimizations to enhance the system performance at the reasonable cost. The purpose of this thesis is twofold. First, the novel architecture supports are provided for the different memory consistency models like: SC, Total Store Ordering (TSO), Partial Store Ordering (PSO), Weak Consistency (WC), Release Consistency (RC) and Protected Release Consistency (PRC) in the NoC-based multi-core (McNoC) systems. The PRC model is proposed as an extension of the RC model which provides additional reordering and relaxation in the memory operations. Second, the scalability analysis of these memory consistency models is performed in the McNoC systems. The architecture supports for these different memory consistency models are provided in the McNoC platforms. Each configurable McNoC platform uses a packet-switched 2-D mesh NoC with deflection routing policy, distributed shared memory (DSM), distributed locks and customized processor interface. The memory consistency models/protocols are implemented in the customized processor interfaces which are developed to integrate the processors with the rest of the system. The realization schemes for the memory consistency models are based on a transaction counter and an an an address ddress ddress ddress ddress ddress ddress stack tacktack-based based based based based based novel approaches.approaches.approaches.approaches. approaches.approaches.approaches.approaches.approaches.approaches. The transaction counter is used in each node of the network to keep track of the outstanding memory operations issued by a processor in the system. The address stack is used in each node of the network to keep track of the addresses of the outstanding memory operations issued by a processor in the system. These hardware structures are used in the processor interface to enforce the required global orders under these different memory consistency models. The realization scheme of the PRC model in addition also uses acquire counter for further classification of the data operations as unprotected and protected operations. The scalability analysis of these different memory consistency models is performed on the basis of different workloads which are developed and mapped on the various sized networks. The scalability study is conducted in the McNoC systems with 1 to 64-cores with various applications using different problem sizes and traffic patterns. The performance metrics like execution time, performance, speedup, overhead and efficiency are evaluated as a function of the network size. The experiments are conducted both with the synthetic and application workloads. The experimental results under different application workloads show that the average execution time under the relaxed memory consistency models decreases relative to the SC model. The specific numbers are highly sensitive to the application and depend on how well it matches to the architectures. This study shows the performance improvement under the relaxed memory consistency models over the SC model that is dependent on the computation-to-communication ratio, traffic patterns, data-to-synchronization ratio and the problem size. The performance improvement of the PRC and RC models over the SC model tends to be higher than 50% as observed in the experiments, when the system is further scaled up. / <p>QC 20130204</p>

In this thesis, we study parallelization of multiplayer games using software Transactional Memory (STM) support. We show that STM provides not only ease of programming, but also better scalability than achievable with state-of-the-art lock-based programming for this realistic high impact application. We evaluate and compare two parallel implementations of a simplified version (named SynQuake) of the popular game Quake. While in STM SynQuake support for maintaining consistency of each potentially complex game action is automatic, conservative locking of surrounding objects within a bounding-box for the duration of the game action is inherently needed in lock-based SynQuake. This leads to higher scalability of STM SynQuake versus lock-based SynQuake due to increased false sharing in the latter. Task assignment to threads has a second-order effect on scalability of STM-SynQuake, impacting the application's true sharing patterns. We show that a locality-aware task assignment provides the best trade-off between load balancing and conflict reduction.

Concept discovery is the problem of finding definitions of target relation in terms or other relation given as a background knowledge. Inductive Logic Programming (ILP)-based and graph-based approaches are two competitors in concept discovery problem. Although ILP-based systems have long dominated the area, graph-based systems have recently gained popularity as they overcome certain shortcomings of ILP-based systems. While having applications in numerous domains, ILP-based concept discovery systems still sustain scalability and efficiency issues. These issues generally arose due to the large search spaces such systems build. In this work we propose memoization-based and parallelization-based methods that modify the search space construction step and the evaluation step of ILP-based concept discovery systems to overcome these problem. In this work we propose three memoization-based methods, called Tabular CRIS, Tabular CRIS-wEF, and Selective Tabular CRIS. In these methods, basically, evaluation queries are stored in look-up tables for later uses. While preserving some core functions in common, each proposed method improves e_ciency and scalability of its predecessor by introducing constraints on what kind of evaluation queries to store in look-up tables and for how long. The proposed parallelization method, called pCRIS, parallelizes the search space construction and evaluation steps of ILP-based concept discovery systems in a data-parallel manner. The proposed method introduces policies to minimize the redundant work and waiting time among the workers at synchronization points. Graph-based approaches were first introduced to the concept discovery domain to handle the so called local plateau problem. Graph-based approaches have recently gained more popularity in concept discovery system as they provide convenient environment to represent relational data and are able to overcome certain shortcomings of ILP-based concept discovery systems. Graph-based approaches can be classified as structure-based approaches and path-finding approaches. The first class of approaches need to employ expensive algorithms such as graph isomorphism to find frequently appearing substructures. The methods that fall into the second class need to employ sophisticated indexing mechanisms to find out the frequently appearing paths that connect some nodes in interest. In this work, we also propose a hybrid method for graph-based concept discovery which does not require costly substructure matching algorithms and path indexing mechanism. The proposed method builds the graph in such a way that similar facts are grouped together and paths that eventually turn to be concept descriptors are build while the graph is constructed.

In this thesis, we study parallelization of multiplayer games using software Transactional Memory (STM) support. We show that STM provides not only ease of programming, but also better scalability than achievable with state-of-the-art lock-based programming for this realistic high impact application. We evaluate and compare two parallel implementations of a simplified version (named SynQuake) of the popular game Quake. While in STM SynQuake support for maintaining consistency of each potentially complex game action is automatic, conservative locking of surrounding objects within a bounding-box for the duration of the game action is inherently needed in lock-based SynQuake. This leads to higher scalability of STM SynQuake versus lock-based SynQuake due to increased false sharing in the latter. Task assignment to threads has a second-order effect on scalability of STM-SynQuake, impacting the application's true sharing patterns. We show that a locality-aware task assignment provides the best trade-off between load balancing and conflict reduction.

The Service-Oriented Architecture (SOA) has become one of the most popular approaches to building large-scale network applications. The web service technologies are de facto the default implementation for SOA. Simple Object Access Protocol (SOAP) is the key and fundamental technology of web services. Service composition is a way to deliver complex services based on existing partner services. Service orchestration with the support of Web Services Business Process Execution Language (WSBPEL) is the dominant approach of web service composition. WSBPEL-based service orchestration inherited the issue of interoperability from SOAP, and it was furthermore challenged for performance, scalability, reliability and modifiability. I present an architectural approach for service composition in this thesis to address these challenges. An architectural solution is so generic that it can be applied to a large spectrum of problems. I name the architectural style RESTful Service Composition (RSC), because many of its elements and constraints are derived from Representational State Transfer (REST). REST is an architectural style developed to describe the architectural style of the Web. The Web has demonstrated outstanding interoperability, performance, scalability, reliability and modifiability. RSC is designed for service composition on the Internet. The RSC style is composed on specific element types, including RESTful service composition client, RESTful partner proxy, composite resource, resource client, functional computation and relaying service. A service composition is partitioned into stages; each stage is represented as a computation that has a uniform identifier and a set of uniform access methods; and the transitions between stages are driven by computational batons. RSC is supplemented by a programming model that emphasizes on-demand function, map-reduce and continuation passing. An RSC-style composition does not depend on either a central conductor service or a common choreography specification, which makes it different from service orchestration or service choreography. Four scenarios are used to evaluate the performance, scalability, reliability and modifiability improvement of the RSC approach compared to orchestration. An RSC-style solution and an orchestration solution are compared side by side in every scenario. The first scenario evaluates the performance improvement of the X-Ray Diffraction (XRD) application in ScienceStudio; the second scenario evaluates the scalability improvement of the Process Variable (PV) snapshot application; the third scenario evaluates the reliability improvement of a notification application by simulation; and the fourth scenario evaluates the modifiability improvement of the XRD application in order to fulfil emerging requirements. The results show that the RSC approach outperforms the orchestration approach in every aspect.