Back to search

Athens University Economics and Business (AUEB)

Fast and efficient predictions in big data systems

Abstract

dc:description

Over the past decade, there has been an exponential surge in the volume of data produced. This data originates from diverse sources, including large-scale applications such as popular social media platforms, and information-sensing devices like mobiles, software logs, cameras, microphones, and wireless sensor networks. This influx of data has fostered the development of applications designed to aid users in daily tasks, including recommendation systems (e.g., suggesting a movie or forecasting traffic patterns). Such applications often necessitate substantial computing resources due to memory-intensive operations, especially when handling data in the GB or TB scale. Distributed big data frameworks, such as Apache Spark and Apache Flink, have become essential for processing enormous volumes of data. While they offer scalable and low-latency data processing by distributing computation across multiple components, their full potential is often hindered by complex challenges, including the critical task of determining the appropriate amount of resources needed to run applications effectively. Another significant concern is reducing the considerable energy consumption of these large clusters. Addressing these challenges is crucial for leveraging the true capacity of big data frameworks. This thesis introduces practical methodologies to tackle these challenges, focusing on efficient job scheduling, accurate prediction models, event detection systems, and energy-efficient resource utilization. This thesis introduces practical methodologies to tackle these challenges: First, we investigate the intricate problem of scheduling real-time jobs on big data frameworks that utilize the MapReduce paradigm. Alongside this, we delve into creating accurate prediction models for gauging application execution time. Addressing the complexities of execution environment heterogeneity, skewed data blocks, real-time application demands, and limited profiling runs, we propose a set of scheduling algorithms and a pioneering profiling framework. Subsequently, we address the challenge of job scheduling with the aim of fulfilling performance metrics while simultaneously conserving cluster resources and energy. Then, We present a system that harnesses traffic data from Dublin city sensors. Using an advanced Matrix Factorization technique, this system is adept at predicting city-wide traffic patterns in real-time. Lastly, we detail a system aimed at delivering recommendations and predictions to users rapidly and accurately. A significant highlight is a system that leverages data from Foursquare, employing a Tensor Factorization technique to recommend Points of Interest based on users' preferences. Empirical evaluations on real-world datasets underscore the practicality and superiority of our proposed methodologies. These findings not only demonstrate our methods' efficiency in job scheduling, traffic prediction, and recommendation systems but also highlight their potential to significantly advance the capabilities of distributed processing systems such as Apache Spark and Apache Flink.

Degree

thesis:*
Grantor dc:publisher
Athens University Economics and Business (AUEB)
Year dc:date
2023

Author and committee

dc:creator, dc:contributor.*
Authors dc:creator
  • Maroulis, Efstathios
  • Μαρούλης, Ευστάθιος

Subjects

dc:subject × 14

Rights

Language dc:language
eng

Identifiers

dc:identifier.*
Identifier
10.12681/eadd/55398

Chain of custody

source
Harvested from
Greek National Archive of PhD Theses
Base URL
phdtheses.ekt.gr/eadd_oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Maroulis, Efstathios; Μαρούλης, Ευστάθιος. Fast and efficient predictions in big data systems. Athens University Economics and Business (AUEB), 2023. http://hdl.handle.net/10442/hedi/55398