Athens University Economics and Business (AUEB)
Fast and efficient predictions in big data systems
Abstract
dc:descriptionOver the past decade, there has been an exponential surge in the volume of data produced. This data originates from diverse sources, including large-scale applications such as popular social media platforms, and information-sensing devices like mobiles, software logs, cameras, microphones, and wireless sensor networks. This influx of data has fostered the development of applications designed to aid users in daily tasks, including recommendation systems (e.g., suggesting a movie or forecasting traffic patterns). Such applications often necessitate substantial computing resources due to memory-intensive operations, especially when handling data in the GB or TB scale. Distributed big data frameworks, such as Apache Spark and Apache Flink, have become essential for processing enormous volumes of data. While they offer scalable and low-latency data processing by distributing computation across multiple components, their full potential is often hindered by complex challenges, including the critical task of determining the appropriate amount of resources needed to run applications effectively. Another significant concern is reducing the considerable energy consumption of these large clusters. Addressing these challenges is crucial for leveraging the true capacity of big data frameworks. This thesis introduces practical methodologies to tackle these challenges, focusing on efficient job scheduling, accurate prediction models, event detection systems, and energy-efficient resource utilization. This thesis introduces practical methodologies to tackle these challenges: First, we investigate the intricate problem of scheduling real-time jobs on big data frameworks that utilize the MapReduce paradigm. Alongside this, we delve into creating accurate prediction models for gauging application execution time. Addressing the complexities of execution environment heterogeneity, skewed data blocks, real-time application demands, and limited profiling runs, we propose a set of scheduling algorithms and a pioneering profiling framework. Subsequently, we address the challenge of job scheduling with the aim of fulfilling performance metrics while simultaneously conserving cluster resources and energy. Then, We present a system that harnesses traffic data from Dublin city sensors. Using an advanced Matrix Factorization technique, this system is adept at predicting city-wide traffic patterns in real-time. Lastly, we detail a system aimed at delivering recommendations and predictions to users rapidly and accurately. A significant highlight is a system that leverages data from Foursquare, employing a Tensor Factorization technique to recommend Points of Interest based on users' preferences. Empirical evaluations on real-world datasets underscore the practicality and superiority of our proposed methodologies. These findings not only demonstrate our methods' efficiency in job scheduling, traffic prediction, and recommendation systems but also highlight their potential to significantly advance the capabilities of distributed processing systems such as Apache Spark and Apache Flink.
Degree
thesis:*- Grantor dc:publisher
- Athens University Economics and Business (AUEB)
- Year dc:date
- 2023
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Maroulis, Efstathios
- Μαρούλης, Ευστάθιος
Subjects
dc:subject × 14- Κατανεμημένα συστήματα
- Συστήματα προτάσεων βάσει περιεχομένου
- Ενεργειακά αποδοτικός χρονοπρογραμματισμός
- Συστήματα ανίχνευσης συμβάντων
- Distributed systems
- Recommendation systems
- Εnergy efficient scheduling
- Event detection systems
- Επιστήμη Ηλεκτρονικών Υπολογιστών και Πληροφορική
- Φυσικές Επιστήμες
- Επιστήμη ηλεκτρονικών υπολογιστών
- Computer and Information Sciences
- Natural Sciences
- Computer Science
Rights
- Language dc:language
- eng
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/55398