Back to search

University of Sheffield

Navigating Multimodal Complexity: Advances in Model Design, Dataset Creation, and Evaluation Techniques

Abstract

dc:description.abstract

Ibn Sina, a philosopher of 11th-century Persia, wrote of a `Floating Man'. This man is floating through a void, without the use of his sight or touch or any of the senses which make us human. Yet as he has a human brain this man, according to Ibn Sina, is capable of imagining and reasoning with the capabilities of any other person. With the development of Large Language Models the field of Artificial Intelligence has come close to making a `Floating Man' - or at least making a `Floating Man' with memories of more books than exist in the wildest dreams of the librarians of Alexandria or Oxford. In this thesis, we question if the `floating man' of AI could benefit from more of his senses, reasoning that as humans a great deal of our experience is multimodal. Our research aims to address the limitations of current NLP models that heavily rely on textual information, often at the expense of multimodal cues. Such errors highlight the critical need for multimodal approaches in many applications, of which we study Visual Question Answering, Citation Recommendation, and Eye-Tracking Prediction, where text alone can lead to biased, harmful, or simply incorrect outcomes, such as mistaking a metal table for one made of wood due to textual biases. Through our research, we aim to show the potential for Multimodality in enriching the capabilities of Artificial Intelligence.

Degree

thesis:*
Name dc:type.qualificationname
Ph.D
Level dc:type.qualificationlevel
doctoral
Grantor dc:publisher.institution
University of Sheffield
Year dc:date.issued
2024

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Vickers, Peter George Jarvis
Advisors dc:contributor.advisor
  • Aletras, Nikos
  • Barrault, Loïc

Rights

Language dc:language.dcterms_iso639_2
eng

Identifiers

dc:identifier.*
Dc Identifier Grantnumber
EP/S023062/1

Chain of custody

source
Harvested from
White Rose University Consortium
Base URL
etheses.whiterose.ac.uk/cgi/oai2
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
related terms
citation

Vickers, Peter George Jarvis. Navigating Multimodal Complexity: Advances in Model Design, Dataset Creation, and Evaluation Techniques. doctoral thesis, University of Sheffield, 2024.