Aristotle University Of Thessaloniki (AUTH)
Massively parallel unsupervised anomaly detection in data streams
Abstract
dc:descriptionOutlier detection on data streams has seen a rise in popularity over the past years due to the advances of Internet of Things devices. Nevertheless, a distributed solution for distance-based outlier detection, which is one of the most widely used categories, has not been proposed for continuous data streams. This thesis proposes a solution to this problem, along with providing an efficient explanation for each outputted outlier, in contrast to the output of normal distance-based detectors. Additionally, it studies the self-adaptive problem for workload balancing on distributed streams and proposes a mechanism that works on the fly without any system interruptions.More specifically, the rationale behind the parallelization of the state-of-the-art centralized distance-based outlier detection algorithms for streams is presented along with the decisions for the partitioning phase of the ingested streams. The proposed techniques show significant improvements over their centralized counter-parts as well as a high throughput when processing an intense stream.Moreover, an explanatory labeling technique is proposed that transforms the binary output of a distance-based detector into an insightful result for the end user. By combining the explanatory process with a subspace exploration technique, the results can be further enhanced with a visualized output and insights into the subspaces and the features of the data stream that mostly contribute to or capture the outlierness. Additionally, the problem of adapting the workload of the partitioning phase has been investigated and a self-adaptive mechanism has been proposed. The mechanism uses metadata from the outlier detection phase to decide whether the value-based partitioning needs to be adapted on the fly. The process is transparent to the user and does not interrupt the system or incur any overhead on the process of outlier detection in real-time.Finally, the set of proposed techniques has been combined into a single complete open-source engine, called PROUD, that allows for distributed distance-based outlier detection for continuous and intense data streams out of the box. The solution is built on top of Apache Flink and supported by secondary open-source frameworks providing numerous extensibility points where the user can implement new techniques and algorithms.
Degree
thesis:*- Grantor dc:publisher
- Aristotle University Of Thessaloniki (AUTH)
- Year dc:date
- 2022
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Toliopoulos, Theodoros
- Τολιόπουλος, Θεόδωρος
Subjects
dc:subject × 14- Εύρεση ακρότατων σημείων με βάση την απόσταση
- Επεξεργασία έντονων ροών δεδομένων
- Επεξήγηση αποτελεσμάτων
- Προσαρμογή φόρτου εργασίας
- Distance-based outlier detection
- Big data stream processing
- Explainable outliers
- Adaptive workload
- Επιστήμη Ηλεκτρονικών Υπολογιστών και Πληροφορική
- Φυσικές Επιστήμες
- Πληροφοριακά συστήματα
- Computer and Information Sciences
- Natural Sciences
- Information Systems
Rights
- Language dc:language
- eng
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/52686