Back to search

Georgia Institute of Technology

On Training, Inference, and Sample Efficiencies of Language Models

Abstract

dc:description.abstract

Large language models have demonstrated superior performance in various natural language processing tasks such as machine translation, natural language understanding, and natural language generation. However, despite the recent developments, language models still face critical challenges. In this thesis, we investigate efficient training and inference algorithms. We also investigate the sample efficiency of training language models. In Chapter 2, we improve training efficiency of sparsely activated models by proposing a novel Mixture-of-Experts architecture. In Chapter 3, we propose state space augmented Transformer models, facilitating efficient modeling of long sequences. In Chapter 4, we target for inference efficiency of pre-trained language models. Specifically, we propose a knowledge distillation algorithm which adapts a pre-trained model into a Mixture-of-Experts model. In Chapter 5, we design a label efficient self-training algorithm. Specifically, we integrate differentiable teacher models into the conventional teacher-student self-training framework.

Degree

thesis:*
Level thesis:degree_level
Doctoral
Department dc:contributor.department
Industrial and Systems Engineering
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2023

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Zuo, Simiao
Advisor dc:contributor.advisor
  • Zhao, Tuo
Committee members dc:contributor.committeemember
  • Zhang, Chao
  • Mei, Yajun
  • Wu, Anqi
  • Liu, Xiaodong

Subjects

dc:subject × 1

Rights

Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/71990

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Zuo, Simiao. On Training, Inference, and Sample Efficiencies of Language Models. Doctoral thesis, Georgia Institute of Technology, 2023. https://hdl.handle.net/1853/71990