University of South Florida
EDGE-STREAMER: A Lightweight, Self-Supervised Event Segmentation Model
Abstract
dc:descriptionEvent segmentation is the practice of autonomously detecting the boundaries of semantically connected sequences of actions within a video. It is connected to many areas of computer vision, including action recognition and event understanding. Nearly all current work requires the entire video to be stored in memory to be processed in multiple passes. This takes up valuable resources, increases processing time, prevents the use of low-power, low-memory devices, and precludes long-form content and live videos from undergoing event segmentation. This thesis introduces EDGE-STREAMER, a real-time, lightweight, self-supervised, transformer architecture capable of performing event segmentation in a single pass. EDGE-STREAMER uses techniques from reinforcement learning models to store representative information about the past to predict events, rather than the entire video. It takes advantage of weight sharing to reduce model size, further reducing its memory footprint and its training requirements. During training the model processes unlabeled video data in a single pass, producing event boundaries in real time. On Epic-Kitchens EDGE-STREAMER achieves the same MoF performance as STREAMER within 2\%. On an NVIDIA Jetson Orin Nano, it sustains over 16 fps, showing that it is capable of real-time performance on an edge device. The contributions of this thesis are 1. Four low-memory event segmentation architectures. 2. A thorough analysis comparing the accuracies of EDGE-STREAMER to the state-of-the-art. 3. Experimental data showing the viability of EDGE-STREAMER on edge hardware. These results open the door for embedded device-based cameras, such as those used for surveillance, autonomous drones, and wearables, and provide possibilities for future work, including multi-modal streaming and embedded-device continual learning.
Degree
thesis:*- Name thesis:degree_name
- MS in Computer Engineering (M.S.C.P.)
- Level thesis:degree_level
- Thesis
- Discipline thesis:degree_discipline
- Computer Science and Engineering
- Grantor
- University of South Florida
- Year dc:date
- 2025
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Miller, Lucas
- Contributors dc:contributor
-
- Sudeep Sarkar, Ph.D.
- Shaun Canavan, Ph.D.
- Mauricio Pamplona Segundo, Ph.D.
Subjects
dc:subject × 6Identifiers
dc:identifier.*- Repository record dc:identifier
- https://digitalcommons.usf.edu/etd/10887