Back to search

University of Sheffield

Understanding how Language Models Learn Under Constrained or Noisy Settings

Abstract

dc:description.abstract

Transformer-based Language models (LMs) have become the foundation of modern Natural Language Processing (NLP), demonstrating strong performance across various tasks. Typically trained on massive datasets with curated objectives and clean supervision, these models are increasingly being deployed in real-world settings where data is noisy, incomplete, or otherwise imperfect. Understanding what drives their learning and how robust they are to such conditions is essential. This thesis investigates how different training signals, namely pre-training objectives, input structure, and instruction quality, affect the generalization behavior of LMs. Through three empirical studies, it examines model performance when traditional assumptions are relaxed or deliberately challenged. The first study explores the role of the pre-training objective, comparing linguistically intuitive objectives with arbitrary or non-linguistic alternatives. Surprisingly, even non-intuitive objectives yield models that acquire linguistic knowledge, suggesting that architecture and data distribution may matter more than the objective’s semantic alignment. The second study focuses on the internal structure of input tokens. Motivated by psycholinguistic findings, we test whether models can learn effectively from inputs with only partial character information. Results show that models remain robust, even with severely reduced inputs, indicating a surprising resilience to information loss. The final study shifts to instruction-tuning. Contrary to the common assumption that clean, well-formed prompts are necessary for generalization, we find that training with noisy or perturbed instructions can often improve both robustness and performance. This suggests that strategically applied noise can act as a form of regularization. Together, these findings deepen our understanding of how LMs learn under constrained or imperfect conditions. They show that models can be not only effective when trained on clean, large-scale data but also surprisingly resilient, and at times improved, when faced with noisy or incomplete signals. This thesis offers practical insights for building robust, data-efficient language models suitable for real-world deployment.

Degree

thesis:*
Name dc:type.qualificationname
Ph.D
Level dc:type.qualificationlevel
doctoral
Grantor dc:publisher.institution
University of Sheffield
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Alajrami, Ahmed J S
Advisor dc:contributor.advisor
  • Aletras, Nikolaos
ORCID person:identifier.orcid

Rights

Language dc:language.dcterms_iso639_2
eng

Identifiers

dc:identifier.*
Dc Identifier Grantnumber
EP/S023062/1

Chain of custody

source
Harvested from
White Rose University Consortium
Base URL
etheses.whiterose.ac.uk/cgi/oai2
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
related terms
citation

Alajrami, Ahmed J S. Understanding how Language Models Learn Under Constrained or Noisy Settings. doctoral thesis, University of Sheffield, 2025.