Back to search

University of Sheffield

Robust Audiovisual Active Speaker Detection for Egocentric Recordings - Leveraging Contextual Information and Cross-Modal Biometric Associations

Abstract

dc:description.abstract

The task of Audiovisual Active Speaker Detection (ASD) involves determining when a visible person in a video is speaking by jointly analysing audiovisual cues indicative of speech. Conventional ASD systems heavily rely on modelling the cross-modal synchronicity between audio-visual cues that arise during periods of active speech. This involves modelling finegrained articulatory motion in the video signal and detecting speech in the audio signal. In egocentric recordings captured from wearable devices, rapid camera motion, partial occlusions, ambient noise, and computational resource constraints degrade the effectiveness of conventional synchronisation-based models. This thesis explores how contextual and cross-modal biometric information can compensate for these limitations, advancing lightweight ASD suitable for real-world, on-device operation. Rather than relying solely on frame-level alignment between facial motion and speech, three complementary strategies are developed that progressively expand the contextual scope of ASD modelling. First, full-scene embeddings derived from transformer-based architectures are introduced to capture the global contextual information provided by full-scene images, allowing baseline ASD models to exploit priors present in the full-scene semantics. Second, speaker-aware conditioning is achieved by proposing a system which extracts and compares vocal-identity embeddings, enabling disambiguation of visually degraded scenarios. Finally, a synchronisation-free framework based on cross-modal Face-Voice Association (FVA) is proposed, where shared biometric embeddings of faces and voices attribute speech to the relevant visible identity implicitly. Together, these contributions demonstrate that contextual reasoning and cross-modal biometric correspondence can substantially improve ASD robustness under egocentric conditions while preserving parameter efficiency. The resulting models achieve competitive or superior performance to heavier synchronisation-based systems on large-scale benchmarks. More broadly, the findings highlight the potential of context-enriched multimodal representations as a foundation for future wearable and Augmented Reality (AR)-based audiovisual diarisation systems.

Degree

thesis:*
Name dc:type.qualificationname
Ph.D
Level dc:type.qualificationlevel
doctoral
Grantor dc:publisher.institution
University of Sheffield
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Clarke, Jason
Advisors dc:contributor.advisor
  • Goetze, Stefan
  • Gotoh, Yoshihiko
ORCID person:identifier.orcid

Rights

Language dc:language.dcterms_iso639_2
eng

Identifiers

dc:identifier.*
Dc Identifier Grantnumber
EP/S023062/

Chain of custody

source
Harvested from
White Rose University Consortium
Base URL
etheses.whiterose.ac.uk/cgi/oai2
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
related terms
citation

Clarke, Jason. Robust Audiovisual Active Speaker Detection for Egocentric Recordings - Leveraging Contextual Information and Cross-Modal Biometric Associations. doctoral thesis, University of Sheffield, 2026.