Georgia Institute of Technology
On Training, Inference, and Sample Efficiencies of Language Models
Abstract
dc:description.abstractLarge language models have demonstrated superior performance in various natural language processing tasks such as machine translation, natural language understanding, and natural language generation. However, despite the recent developments, language models still face critical challenges. In this thesis, we investigate efficient training and inference algorithms. We also investigate the sample efficiency of training language models. In Chapter 2, we improve training efficiency of sparsely activated models by proposing a novel Mixture-of-Experts architecture. In Chapter 3, we propose state space augmented Transformer models, facilitating efficient modeling of long sequences. In Chapter 4, we target for inference efficiency of pre-trained language models. Specifically, we propose a knowledge distillation algorithm which adapts a pre-trained model into a Mixture-of-Experts model. In Chapter 5, we design a label efficient self-training algorithm. Specifically, we integrate differentiable teacher models into the conventional teacher-student self-training framework.
Degree
thesis:*- Level thesis:degree_level
- Doctoral
- Department dc:contributor.department
- Industrial and Systems Engineering
- Grantor dc:publisher
- Georgia Institute of Technology
- Year dc:date.issued
- 2023
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Zuo, Simiao
- Advisor dc:contributor.advisor
-
- Zhao, Tuo
- Committee members dc:contributor.committeemember
-
- Zhang, Chao
- Mei, Yajun
- Wu, Anqi
- Liu, Xiaodong
Subjects
dc:subject × 1Rights
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/71990