University of Piraeus (UNIPI)
Real-Time monitoring of data streams using machine learning techniques for optimizing decision making
Abstract
dc:descriptionThe growth of big data is primarily driven by the increasing digitization of information and the widespread use of data-collecting devices. As technological advancements continue, the volume, variety, and velocity of data generation expand, giving rise to big data analytics aimed at extracting valuable insights from vast datasets. The growing volume of data presents both opportunities and significant challenges. In the realm of Statistical Process Monitoring (SPM), the analysis of high-dimensional data often leads to the “curse of dimensionality”, where data sparsity hinders the detection of meaningful patterns and anomalies. Additionally, monitoring complex relationships among multiple variables requires more advanced methods than traditional univariate approaches. Multivariate Statistical Process Monitoring (MSPM) addresses this need by employing tools such as multivariate control charts, notably the Hotelling’s T² chart, to capture the joint behavior of correlated quality variables. However, implementing these control charts in real-time, high-dimensional settings is particularly difficult due to the computational demands and the need for rapid decision-making when processing large streams of data. A potential solution is to combine traditional MSPM techniques with modern machine learning approaches; even so, this integration poses challenges related to model interpretability, feature selection, and result integration. In this study, a modern and robust method inspired by cluster validity indexing techniques is presented. This method is compared to the traditional multivariate control charts based on the Hotelling T² statistic, as well as to other metrics from the cluster analysis framework, such as the Dunn, Silhouette, Calinski-Harabasz, and Davies-Bouldin indices. Extensive simulation studies demonstrate that the proposed method outperforms existing approaches, particularly in scenarios such as mean drifting and density-related changes, involving data streams with correlated features. At the conclusion, various real-world application scenarios utilizing statistical and machine learning techniques are presented. Ranging from simulations to practical applications, this dissertation seeks to bridge the concepts of statistics and machine learning, with the latter inheriting tools and methodologies from the former.
Degree
thesis:*- Grantor dc:publisher
- University of Piraeus (UNIPI)
- Year dc:date
- 2026
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Skarlatos, Kyriakos
- Σκαρλάτος, Κυριάκος
Subjects
dc:subject × 27- Ροές δεδομένων / Δυναμικά περιβάλλοντα
- Έλεγχος διεργασιών
- Μηχανική μάθηση
- Monte Carlo προσομοίωση
- Συσταδοποίηση
- Λήψη αποφάσεων
- Process monitoring
- Machine Learning Model Classification and Training
- Artificial intelligence algorithms
- Data streaming
- Real time control
- Statistical process monitoring
- Clustering algorithms
- Decision making
- Control charts
- Cluster validity indices
- Hotelling's T2
- Επιστήμη Ηλεκτρονικών Υπολογιστών και Πληροφορική
- Φυσικές Επιστήμες
- Τεχνητή νοημοσύνη
- Μαθηματικά
- Στατιστική και Πιθανότητες
- Computer and Information Sciences
- Natural Sciences
- Artificial Intelligence
- Mathematics
- Statistics and Probability
Rights
- Language dc:language
- eng
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/61310