Back to search

École de technologie supérieure

Elastic edge-based stream processing over Apache Storm

Abstract

dc:description

Apache Storm is a distributed stream processing framework designed to handle continuous data flows, yet its elasticity remains limited when components are deployed across heterogeneous nodes and incoming data rates vary over time. Existing work on Apache Storm does not adequately address how elasticity can be achieved without manual intervention or system restarts. As a result, performance bottlenecks in CPU utilization and bandwidth often persist, increasing end-to-end latency. This thesis introduces an additional layer on top of Apache Storm to enhance its elasticity under distributed and dynamic conditions. The proposed solution addresses both the avoidance and resolution of bottlenecks in inbound bandwidth, outbound bandwidth, and CPU resources, aiming to minimize overall processing latency. Elasticity was achieved through two complementary mechanisms. The first, global adaptation, monitors system performance to decide when and where to create new replicas and when to safely remove underutilized replicas without harming throughput. The second, local adaptation, is implemented at the operator level as a resource-aware load distribution strategy, balancing data across downstream replicas. Together, these mechanisms helped ensure that bottlenecks are avoided as much as possible and effectively resolved when they occur. The system was implemented on Docker for deployment across heterogeneous nodes. Experimental results show that the system can create and remove replicas on the fly, without interrupting execution, while successfully detecting and mitigating CPU and bandwidth bottlenecks. These mechanisms significantly reduce average end-to-end latency and improve overall resource utilization compared to baseline configurations. Moreover, the findings showed that the proposed solution can scale reliably within the experimental setup. This research provides a practical approach to enhancing elasticity in distributed stream processing systems, enabling robust performance in dynamic and heterogeneous environments.

Degree

thesis:*
Name thesis:degree_name
Mémoire de maîtrise électronique
Level thesis:degree_level
masters
Grantor dc:publisher
École de technologie supérieure
Year dc:date
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Shahabadi, Mitra

Subjects

dc:subject × 1

Rights

Language dc:language
en

Identifiers

dc:identifier.*
Identifier
INSERT-YOUR-LIBRARY-ID-HERE3779

Chain of custody

source
Harvested from
ETS (Quebec)
Base URL
espace.etsmtl.ca/cgi/oai2
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Shahabadi, Mitra. Elastic edge-based stream processing over Apache Storm. masters thesis, École de technologie supérieure, 2025.