Back to search

University of Birmingham

Unfolding shared semantic representations in vision and language

Abstract

dc:description.abstract

When we interact with language, the brain effortlessly draws vivid mental images that resemble representations of directly perceived visual scenes. Similarly, research shows that the visual system encodes representations of visual scenes in a manner akin to how embeddings from large language models encode the meaning of a sentence. This implies that vision and language draw on a shared representational system for semantic encoding, but the precise nature of this visuo-linguistic overlap in the brain, as well as its implications for behaviour, remain unknown. The present thesis leveraged similarity judgements of natural scene images and their corresponding sentence captions to explore how vision and language converge in their ability to recognise semantic regularities of the visual world. We found that the visual and linguistic similarity judgements not only converge in the behavioural context but both also predict a remarkably similar network of visually evoked patterns along the mid- and high-level visual regions. Furthermore, we established the behavioural relevance by showing that the perceived dissimilarities in both vision and language effectively predict response times to a visual target primed by a sentence. Upon the observation that linguistic similarity judgements exhibit a higher interindividual variability, we used the participants’ own perceived dissimilarities to make predictions about their priming effect. This revealed that idiosyncratic linguistic dissimilarity is the strongest predictor of participants’ response times. The findings of the present thesis demonstrate that vision and language project their representations onto a shared similarity space, most likely embedded in the representational structure of the visual brain. Along with the observation that language plays a distinct role in eliciting individually relevant representations, these findings may profoundly impact our understanding of the visual system by placing greater emphasis on semantic idiosyncrasies.

Degree

thesis:*
Name dc:type.qualificationname
d_ph
Level dc:type.qualificationlevel
d_ph
Grantor dc:publisher.institution
University of Birmingham
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Simkova, Katerina Marie
ORCID person:identifier.orcid

Subjects

dc:subject × 2

Rights

Language dc:language.dcterms_iso639_2
eng

Chain of custody

source
Harvested from
University of Birmingham
Base URL
etheses.bham.ac.uk/cgi/oai2
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Simkova, Katerina Marie. Unfolding shared semantic representations in vision and language. d_ph thesis, University of Birmingham, 2025.