Back to search

Virginia Tech

SoundBreak: Studying vulnerabilities exposed by Audio Adversarial Attacks on Trimodal Models

Abstract

dc:description.abstract

Foundation models that jointly process audio, visual, and textual inputs have demonstrated strong performance across a wide range of reasoning and generation tasks. Despite these advances, the robustness of such systems against adversarial manipulation remains insufficiently explored. This thesis presents SoundBreak, a comprehensive study of untargeted adversarial attacks that perturb only the audio modality in trimodal audio-video-language models. We introduce six base attack objectives and a combined objective that target different components of the multimodal pipeline, including audio encoder representations, cross-modal attention distributions, internal hidden-state dynamics, and output token likelihoods. Extensive experiments conducted across four state-of-the-art multimodal models and multiple benchmark datasets show that audio-only perturbations are sufficient to induce substantial performance degradation, achieving attack success rates of up to 96%. The combined objective achieves the highest attack success rate, while the encoder-space objective provides the clearest high-effectiveness, low-distortion trade-off, reaching 89.12% ASR with LPIPS 0.08 and SI-SNR -1.77 dB. The lowest-distortion objectives achieve LPIPS 0.06 and SI-SNR around 0 dB, but with lower ASR. Our analysis further reveals that prolonged optimization contributes more significantly to attack effectiveness than increased dataset scale, while transferability across models and audio encoders remains limited. Additionally, automatic speech recognition systems such as Whisper are found to respond primarily to overall distortion magnitude, reaching attack success rates exceeding 97% under high-distortion objectives. Collectively, these findings expose a previously underexplored single-modality vulnerability in multimodal systems and highlight the need for robust defense mechanisms based on cross-modal consistency and agreement verification.

Degree

thesis:*
Name thesis:degree_name
Master of Science
Level thesis:degree_level
masters
Discipline thesis:degree_discipline
Computer Science & Applications
Department dc:contributor.department
Computer Science and#38; Applications
Grantor dc:publisher
Virginia Tech
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Hussain, Aafiya Shamshad
Chair dc:contributor.committeechair
  • Thomas, Christopher Lee
Committee members dc:contributor.committeemember
  • Jia, Ruoxi
  • Yanardag Delul, Pinar

Subjects

dc:subject × 5

Rights

dc:rights
Statement dc:rights
  • In Copyright
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Other
vt_gsexam:46725

Chain of custody

source
Harvested from
Virginia Tech
Base URL
vtechworks.lib.vt.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Hussain, Aafiya Shamshad. SoundBreak: Studying vulnerabilities exposed by Audio Adversarial Attacks on Trimodal Models. masters thesis, Virginia Tech, 2026. https://hdl.handle.net/10919/143646