Back to search

University of Thessaly (UTH)

Human activity recognition under occlusion

Abstract

dc:description

Human activity recognition (HAR) is the task of identifying human behavior based on measurements and observations. It is a challenging computer vision task that has gained significant attention in the research community. HAR approaches can utilize various types of sensors, to capture and analyze human activities. However, wearable sensors are not preferred by users, and sensors placed in the environment can be costly or require interventions. Therefore, low-cost solutions based on camera-based motion capture in 3D space are commonly used. HAR has applications in video surveillance, human-computer/robot interaction, augmented reality, health monitoring, intelligent driving, gaming, animation, and more. Actions and activities are considered complete human motions or sequences of actions forming a meaningful human action, although these terms are mostly used interchangeably, while their recognition from motion data is considered one of the most challenging computer vision problems, involving the recognition of human behavior in images or video sequences. Camera-based HAR approaches are low-cost and perform well in laboratory conditions. However, they face challenges in real-life conditions, including viewpoint and illumination changes, as well as occlusion. Viewpoint variance occurs when the subject is observed from a different viewpoint than during training. Multiple cameras can compensate for this. Illumination changes affect video-based approaches, but depth information can help overcome this challenge. Occlusion, caused by furniture or the presence of other humans, is the most limiting problem. Full occlusion makes HAR impossible, but partial occlusion can still allow recognition depending on the visible skeleton subset. The effect of partial occlusion on HAR accuracy has been studied, showing that it can significantly impact recognition performance, with the extent varying for each activity. This Thesis begins by introducing the datasets that will be used to experimentally evaluate the proposed approaches. We discuss the primary visual sensors commonly used in contemporary computer vision approaches for Human Activity Recognition (HAR), as well as various computer vision libraries that can complement these sensors. We also provide an overview of the research efforts in the broader field of HAR using visual data. Next, we present the theoretical and experimental background for the methodologies that will be discussed in this Thesis. Specifically, we outline the main machine learning algorithms employed, including Convolutional Neural Networks, Recurrent Neural Networks, and Generative Adversarial Networks. Additionally, we describe the programming languages and libraries utilized for implementation, we provide a detailed explanation of the 3D skeletal data used as input for our approaches, and we present the experimental evaluation protocols that align with the existing literature. Subsequently, we commence the presentation of our research with a pseudo-colored image representation for HAR. This representation aims to capture the characteristics of 3D skeletal motion during an activity within a single image. We propose a fusion approach that combines handcrafted and contextual features for HAR. The objective of this approach is to assess whether features extracted from a Convolutional Neural Network trained with pseudo-colored representations of 3D skeletal motion can enhance the performance of traditional approaches. In other words, we investigate whether these features can provide additional cues for activity recognition. Furthermore, we propose an approach to address the issue of occlusion in HAR problems. This approach involves augmenting the training dataset with both occluded and non-occluded samples. We then attempt to overcome the occlusion problem using a deep regression network, which aims to reconstruct missing skeletal joints. Additionally, we explore the use of a Generative Adversarial Network for joint reconstruction. In both reconstruction approaches, classification is performed using a network trained exclusively on non-occluded samples. Finally, we present extensive conclusions regarding all approaches that are presented and we discuss possible research directions towards the continuation of this work.

Degree

thesis:*
Grantor dc:publisher
University of Thessaly (UTH)
Year dc:date
2023

Author and committee

dc:creator, dc:contributor.*
Authors dc:creator
  • Vernikos, Ioannis
  • Βερνίκος, Ιωάννης

Subjects

dc:subject × 16

Rights

Language dc:language
eng

Identifiers

dc:identifier.*
Identifier
10.12681/eadd/55806

Chain of custody

source
Harvested from
Greek National Archive of PhD Theses
Base URL
phdtheses.ekt.gr/eadd_oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Vernikos, Ioannis; Βερνίκος, Ιωάννης. Human activity recognition under occlusion. University of Thessaly (UTH), 2023. http://hdl.handle.net/10442/hedi/55806