Athens University Economics and Business (AUEB)
Deterministic, elastic and real - time processing in the big data era
Abstract
dc:descriptionIn recent years we observe a large increase in the amount of data thatneeds to be analyzed and processed in real-time in a variety ofapplication domains, ranging from traffic monitoring to healthcareinfrastructures and financial processing. In order to processefficiently such high volume of data, novel distributed big dataframeworks such as Apache Storm and Apache Spark have been proposed.Such systems offer scalable and low latency data processing bydistributing the computation between multiple concurrently runningcomponents. However, there exist several important research challengesthat need to be addressed in order to use them in their fullpotential. These challenges include, but are not limited to, providingdeterminism guarantees for the processed tuples, determining thenumber of processing components that should be utilized, handling thedata skewness and efficiently scheduling multiple concurrently runningapplications with real-time response requirements. The purpose of thisthesis is to propose practical methods to address these issues.The first part of the thesis improves big data frameworks by tacklingthe problem of coping with the demands of determinism and meetinglatency constraints. In order to deterministically process thestreaming data, we need mechanisms that synchronize the order in whichtuples are processed by the operators. On the other hand, achievingreal-time response in such a system requires to efficiently handle thetradeoff between determinism and low latency performance. Consideringthe challenges and trade-offs implied by real-time constraints, in thecontext of this thesis a set of systems and methodologies are proposedthat enable the end-users and the system administrators to dynamicallyrelax the determinism guarantees when needed, in order to satisfy thelatency constraints.The second part of the thesis focuses on the problem of schedulingjobs with real-time response time requirements on big data frameworksthat use the MapReduce programming paradigm and also on buildingaccurate prediction models for estimating the execution time of theapplications. The problems are particularly challenging due to theheterogeneity of the execution environment, the load imbalance causedby skewed data blocks, the real-time response demands imposed by theapplications as well as the limited number of available profilingruns. A set of scheduling algorithms and a novel profiling frameworkthat exploits the similarity of the applications' execution plans areproposed for addressing the aforementioned problems. Finally, in thethird part of the thesis, we contribute to the problem of efficientresource management in distributed stream processing systems. Wepropose and develop a novel elasticity mechanism that enables topro-actively determine the degree of parallelism of the big dataframeworks' components and we also consider the applicability ofwell-known load balancing algorithms to further improve the system'sthroughput.Throughout the thesis, our methodologies have been evaluated onreal-world datasets and scenarios. Our experimental results have shownthat our proposed algorithms systematically outperform the competingapproaches and that they constitute practical approaches which can beemployed in widely utilized distributed stream processing systems,like Apache Storm and Apache Spark.
Degree
thesis:*- Grantor dc:publisher
- Athens University Economics and Business (AUEB)
- Year dc:date
- 2018
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Zacheilas, Nikolaos
- Ζαχείλας, Νικόλαος
Subjects
dc:subject × 6Rights
dc:rights- Statement dc:rights
-
- BY
- Language dc:language
- eng
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/44613