Georgia Institute of Technology
Contrastive Multiview Coding with Hard Negatives for Multimodal Human Activity Recognition
Abstract
dc:description.abstractHuman activity recognition (HAR) systems have been extensively studied by the vision and ubiquitous computing communities due to their practical applications in daily life, such as smart homes, surveillance, and health monitoring. However, supervised HAR training requires a large amount of annotated data, making it costly and time-consuming. To address this challenge, contrastive self-supervised learning (SSL) based approaches have gained a lot of attention in the HAR community, as they learn from unannotated data. Nevertheless, a remaining challenge in SSL for HAR is the reliance on positive and negative pairs for feature representation learning. While obtaining true positive pairs is relatively easy and can be achieved by augmenting the anchor sample, obtaining true negative pairs is challenging, as negative samples are uniformly selected from a batch, which may correspond to false negative samples with the same label as the anchor class. In this paper, we propose a hard negative sampling method for multimodal HAR by implementing contrastive multiview coding (CMC) with a hard negative sampling loss namely, CMC-HNL. We exploit hard negatives that have different labels from the anchor and are projected nearby in the latent space with an adjustable concentration parameter to balance improved learning from hard negatives and the potential negative impact of approximately correcting false negatives. Through extensive experiments on two multimodal HAR datasets, UTD-MHAD and MMAct, we demonstrate the robustness of our approach for feature representation learning and semi-supervised learning. Our model outperforms the supervised model and state-of-the-art methods.
Degree
thesis:*- Level thesis:degree_level
- Masters
- Department dc:contributor.department
- Electrical and Computer Engineering
- Grantor dc:publisher
- Georgia Institute of Technology
- Year dc:date.issued
- 2023
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Choi, Hyeongju
- Advisor dc:contributor.advisor
-
- Essa, Irfan
- Committee members dc:contributor.committeemember
-
- Yeo, Woon-hong
- Heck, Larry
Subjects
dc:subject × 5Rights
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/75124