Back to search

Aristotle University Of Thessaloniki (AUTH)

Resource allocation and quality management in geo-distributed workflows

Abstract

dc:description

As the amount of data being stored in geographically distributed data centers is growing and the need to analyze them in a fast manner remains, the option to aggregate them in a central location before processing them becomes significantly inefficient. Therefore, the need to analyze data closer to where they are produced arises. Numerous research efforts have been made to determine the optimal placement of such analysis, usually depicted as a Directed Acyclic Graph (DAG) consisting of multiple steps, to minimize various performance objectives. Additionally, the emergence of Internet of Things and edge computing paradigm where multiple sensors produce massive amounts of data that need to be analyzed in real-time using near-by devices, rather than the cloud, incurs extra challenges such as severe hardware and network heterogeneity, privacy and security constraints. Even after an optimized resource allocation plan is devised, the latency may still be high due to the amount of data being analyzed. Thus, there is a need for development of techniques that reduce this size while also account for the quality-of-result, through measuring the different dimensions of data quality, such as completeness. This thesis aims to address these challenges. First, a multi-objective resource allocation solution for geo-distributed analytics is proposed. The developed techniques take into consideration the heterogeneity of the network links, achieving an average 44.31% reduction of the total running time and at the same time a 50.12% reduction of traffic compared to the state-of- the-art solution. Moreover, a prototype on Apache Spark showcases the performance gains in a real-world scenario. Next, techniques for an edge computing environment that consider the heterogeneity of edge devices and network links, and privacy constraints while incorporating data quality verification tasks, are proposed. Evaluation of the proposed techniques through simulations shows improvements in terms of an objective function that considers latency and extent of quality checks, up to 2.56X compared to state-of-the-art solutions. In addition, the proposed solution reduces latency by up to 8X in 86.9% of cases compared to the default Apache Storm scheduler. Moreover, an empirical study on the performance and scalability benefits that big data frameworks can bring to data quality assessment is conducted. Additionally, the problem of jointly optimizing multiple DAGs is tackled though employing a technique that decides on task placements that efficiently utilize edge devices and finds optimal trade-offs between quality reduction and latency. Evaluation results in Apache Storm show reductions of latency up to 6X with an average speedup of 2.76X compared to the advanced Resource-Aware scheduler. The techniques are also transferred to a data replication problem that further optimizes the applications’ latency, by up to an order of magnitude. Finally, a vision for next-generation, edge-enabled, big data analytics solutions is provided, where the remaining open issues as well as the new challenges posed by the deployment of even more data-intensive applications such as deep learning, are highlighted and a road-map for addressing them is presented, capitalizing also on the results and outcome of this thesis.

Degree

thesis:*
Grantor dc:publisher
Aristotle University Of Thessaloniki (AUTH)
Year dc:date
2023

Author and committee

dc:creator, dc:contributor.*
Authors dc:creator
  • Michailidou, Anna-Valentini
  • Μιχαηλίδου, Άννα-Βαλεντίνη

Subjects

dc:subject × 14

Rights

Language dc:language
eng

Identifiers

dc:identifier.*
Identifier
10.12681/eadd/53516

Chain of custody

source
Harvested from
Greek National Archive of PhD Theses
Base URL
phdtheses.ekt.gr/eadd_oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Michailidou, Anna-Valentini; Μιχαηλίδου, Άννα-Βαλεντίνη. Resource allocation and quality management in geo-distributed workflows. Aristotle University Of Thessaloniki (AUTH), 2023. http://hdl.handle.net/10442/hedi/53516