Back to search

University of Western Macedonia

Acceleration and optimization for real-time multimodal processing on the edge

Abstract

dc:description

Edge processing has emerged as a critical domain in computer vision, driven by the increasing amounts of data sources, over the traditional cloud computing architectures. Cloud computing has significant disadvantages in addressing latency, bandwidth, and security concerns, especially on real-time applications. Edge computing addresses these issues by enabling local data processing, therefore reducing latency, bandwidth usage, and preserving privacy. This paradigm shift empowers advanced computer vision applications such as real-time video analytics, facial recognition, and augmented reality (AR), driving innovation and opening new possibilities. Computer vision aims to bridge the gap between human perception and machine understanding, yet it faces significant challenges, particularly in handling large volumes of visual data. Deep learning (DL) approaches such as the Convolutional Neural Networks (CNNs), have been crucial in modeling pixel relations and recognizing complex patterns. However, the integration of multiple modalities by fusing data from multiple sources is essential for improving accuracy and robustness. This integration poses significant challenges and requires a sophisticated handling and alignment of the diverse data types, especially under the resource constraints of edge devices. This dissertation aims to address these challenges by proposing a novel design methodology for edge processing, leading to an accelerated multimodal framework tailored for these environments. This framework enables the execution of complex and in-depth data processing directly at the source, leveraging novel artificial intelligence (AI) models and optimizations for various applications, such as abnormal event detection, object recognition, proximity assessment, and facial recognition. These approaches demonstrate significant improvements in decision-making capabilities, response times, and overall system performance, even with the limited resources of embedded systems. To mitigate the computational overhead of data preprocessing, we have implemented various computational processing designs in hardware, taking advantage of the parallelism of Field-Programmable Gate Arrays (FPGAs). We introduce a portable VHSIC Hardware Description Language (VHDL) design for color transformation and Sobel edge detection, further improved using High-Level Synthesis (HLS) for increased efficiency. We also present an accelerated noise reduction technique based on image stacking, highlighting the potential of hardware acceleration for edge processing. Additionally, our research investigates the fusion of multimodal sensor data, combining information from various sources such as video, audio, and other sensors. This approach provides a more comprehensive understanding of the environment, enabling even more informed decision-making and improved system performance in complex, real-world scenarios. The effectiveness of this methodology is demonstrated through an extensive evaluation on real-world datasets and deployment scenarios in public transportation, emphasizing passenger safety and real-time decision-making. The results highlight the framework’s versatility and robustness, showcasing its potential to transform diverse applications through the power of edge computing and multimodal data processing.

Degree

thesis:*
Grantor dc:publisher
University of Western Macedonia
Year dc:date
2024

Author and committee

dc:creator, dc:contributor.*
Authors dc:creator
  • Tsiktsiris, Dimitrios
  • Τσικτσίρης, Δημήτριος

Subjects

dc:subject × 18

Rights

Language dc:language
eng

Identifiers

dc:identifier.*
Identifier
10.12681/eadd/57351

Chain of custody

source
Harvested from
Greek National Archive of PhD Theses
Base URL
phdtheses.ekt.gr/eadd_oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Tsiktsiris, Dimitrios; Τσικτσίρης, Δημήτριος. Acceleration and optimization for real-time multimodal processing on the edge. University of Western Macedonia, 2024. http://hdl.handle.net/10442/hedi/57351