Back to search

Aristotle University Of Thessaloniki (AUTH)

Massively parallel unsupervised anomaly detection in data streams

Abstract

dc:description

Outlier detection on data streams has seen a rise in popularity over the past years due to the advances of Internet of Things devices. Nevertheless, a distributed solution for distance-based outlier detection, which is one of the most widely used categories, has not been proposed for continuous data streams. This thesis proposes a solution to this problem, along with providing an efficient explanation for each outputted outlier, in contrast to the output of normal distance-based detectors. Additionally, it studies the self-adaptive problem for workload balancing on distributed streams and proposes a mechanism that works on the fly without any system interruptions.More specifically, the rationale behind the parallelization of the state-of-the-art centralized distance-based outlier detection algorithms for streams is presented along with the decisions for the partitioning phase of the ingested streams. The proposed techniques show significant improvements over their centralized counter-parts as well as a high throughput when processing an intense stream.Moreover, an explanatory labeling technique is proposed that transforms the binary output of a distance-based detector into an insightful result for the end user. By combining the explanatory process with a subspace exploration technique, the results can be further enhanced with a visualized output and insights into the subspaces and the features of the data stream that mostly contribute to or capture the outlierness. Additionally, the problem of adapting the workload of the partitioning phase has been investigated and a self-adaptive mechanism has been proposed. The mechanism uses metadata from the outlier detection phase to decide whether the value-based partitioning needs to be adapted on the fly. The process is transparent to the user and does not interrupt the system or incur any overhead on the process of outlier detection in real-time.Finally, the set of proposed techniques has been combined into a single complete open-source engine, called PROUD, that allows for distributed distance-based outlier detection for continuous and intense data streams out of the box. The solution is built on top of Apache Flink and supported by secondary open-source frameworks providing numerous extensibility points where the user can implement new techniques and algorithms.

Degree

thesis:*
Grantor dc:publisher
Aristotle University Of Thessaloniki (AUTH)
Year dc:date
2022

Author and committee

dc:creator, dc:contributor.*
Authors dc:creator
  • Toliopoulos, Theodoros
  • Τολιόπουλος, Θεόδωρος

Subjects

dc:subject × 14

Rights

Language dc:language
eng

Identifiers

dc:identifier.*
Identifier
10.12681/eadd/52686

Chain of custody

source
Harvested from
Greek National Archive of PhD Theses
Base URL
phdtheses.ekt.gr/eadd_oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Toliopoulos, Theodoros; Τολιόπουλος, Θεόδωρος. Massively parallel unsupervised anomaly detection in data streams. Aristotle University Of Thessaloniki (AUTH), 2022. http://hdl.handle.net/10442/hedi/52686