Back to search

University of British Columbia

Interpreting multimodal speech-vision models with sparse autoencoders

Abstract

dc:description

Vision-language models are a form of multimodal AI that encode language and visual information jointly for a range of downstream tasks, however interpretation of how they understand the input across modalities remains difficult. This work uses Sparse Autoencoders (SAEs) to interpret a self-supervised audio-visual model, focusing on spoken language rather than text as the linguistic modality. We train and compare four SAE architectures (Standard, JumpReLU, BatchTopK, and Matryoshka) on audio-only, image-only, and joint embeddings, and evaluate them along a sparsity-fidelity tradeoff. Qualitative analysis shows that individual SAE dimensions recover concepts at multiple levels of granularity, from single words and semantic categories to phrase-level and position-sensitive structure in the audio stream. Some dimensions capture near-singular lexical items, while others generalize across semantically or phonetically related words. Audio-focused SAEs are consistently less monosemantic than image-focused ones, and Matryoshka models show lower average monosemanticity than JumpReLU models, indicating greater feature entanglement. To test whether concepts are causally shared across modalities, we introduce an image-attenuated word segmentation task: suppressing an SAE dimension in the image branch and propagating this intervention to the audio embedding. Ablating a dimension in the image modality degrades the localization of the corresponding spoken word relative to controls, but the effect is small unless the analysis is restricted to the subset of dimensions with genuinely multimodal activation, at which point the effect becomes substantially clearer and more consistent, particularly for JumpReLU models. Audio ablation paradoxically produces little change in Matryoshka models, which we attribute to hierarchical entanglement across dimensions. These results indicate that our studied VLMs develop some modality-agnostic conceptual representations through unsupervised statistical learning, but that a persistent modality gap separates audio and image encodings of the same concept.

Degree

thesis:*
Name thesis:degree_name
Master of Arts - MA
Level thesis:degree_level
master's
Discipline thesis:degree_discipline
Linguistics
Grantor dc:publisher
University of British Columbia
Year dc:date
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Diep, Brian
Contributors dc:contributor
  • Zhu, Jian

Rights

dc:rights
Statement dc:rights
  • Attribution-NonCommercial-NoDerivatives 4.0 International
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/95495

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Diep, Brian. Interpreting multimodal speech-vision models with sparse autoencoders. master's thesis, University of British Columbia, 2026. http://hdl.handle.net/2429/95495