Back to search

University of Sheffield

Exploring Efficient Methods for Transformer-based Foundation Language Models

Abstract

dc:description.abstract

Transformer-based language models have achieved huge success in various natural language processing tasks, but their efficiency remains a critical challenge. The core limitations stem from the computationally intensive dot-product attention mechanism, which scales quadratically with input length, and the increasing memory footprint associated with large model sizes and extensive vocabularies. These issues hinder their deployment on resource-constrained devices and limit their ability to process long contexts. This thesis explores novel methods to enhance the efficiency of Transformer-based models, focusing on architectural modifications during the pre-training stage. It is presented as a collection of three peer-reviewed publications, each addressing a specific component of the Transformer architecture. The first work introduces a parameter-efficient embedding layer that uses a hashing function to support an unlimited vocabulary with a fixed-size embedding matrix. This approach effectively breaks the rigid, one-to-one mapping between tokens and embeddings, significantly reducing memory consumption. The second work tackles the parameter redundancy within the multi-head attention mechanism. It proposes a more efficient alternative that uses a single shared projection matrix and multiple head embeddings, substantially reducing the number of attention-related parameters while preserving model performance. The final work systematically analyses the key design principles of the dot-product attention mechanism. The findings provide insights into what makes attention so effective and offer a foundation for developing more streamlined and efficient attention mechanisms in the future. This thesis demonstrates that fine-grained architectural modifications during pre-training can yield substantial improvements in model efficiency. The proposed methods are orthogonal to existing post-training compression techniques, providing a complementary approach to creating more scalable and deployable language models. The findings collectively contribute to a deeper understanding of the core components of Transformer models and pave the way for designing next-generation language models that are both powerful and efficient.

Degree

thesis:*
Name dc:type.qualificationname
Ph.D
Level dc:type.qualificationlevel
doctoral
Grantor dc:publisher.institution
University of Sheffield
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Xue, Huiyin
Advisor dc:contributor.advisor
  • Aletras, Nikolaos
ORCID person:identifier.orcid

Rights

Language dc:language.dcterms_iso639_2
eng

Chain of custody

source
Harvested from
White Rose University Consortium
Base URL
etheses.whiterose.ac.uk/cgi/oai2
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
related terms
citation

Xue, Huiyin. Exploring Efficient Methods for Transformer-based Foundation Language Models. doctoral thesis, University of Sheffield, 2025.