University of British Columbia
Towards understanding self-attention and batch normalization : optimization and margin dynamics in neural networks
Abstract
dc:descriptionDeep neural networks have achieved transformative success in numerous fields such as computer vision, natural language processing, and robotics, largely due to their ability to capture complex patterns from large datasets. Among the innovations driving this success, the transformers mechanism has played a pivotal role by overcoming the limitations of earlier models, like recurrent neural networks. Despite the empirical success, our theoretical understanding of transformers and their core, self-attention mechanism remains incomplete, especially with regard to their optimization and generalization behaviors. Previous research has predominantly focused on single-head attention, thereby limiting insights into the impact of overparameterization in more complex models. Motivated by the demonstrated benefits of overparameterization in training fully-connected networks, we explore the potential optimization and generalization advantages of employing multiple attention heads. Specifically, we derive convergence and generalization guarantees for gradient descent training of a single-layer multi-head self-attention model, under a suitable realizability condition on the data. Additionally, we identify initialization conditions that ensure the realizability assumption is met and verify that these conditions hold in a simple but practical data model. Our findings offer insights that could be extended to various data models and architectural settings. Building upon these insights, we also examine the influence of other architectural components on training dynamics, focusing specifically on normalization. To this end, we study batch normalization, a widely-used technique that has proven to accelerate training and improve generalization in deep neural networks. Despite its popularity, our understanding of its underlying mechanisms remains limited, and we believe that uncovering its impact could further help extend our results to the other normalization variants across different neural network architectures. We investigate the impact of batch normalization on a simple two-layer neural network and show that when training with gradient descent on a simple data model, the network parameters converge to a solution with uniform margin directions. Furthermore, we derive finite-time rates for the evolution of both the loss and the weights during the training process.
Degree
thesis:*- Name thesis:degree_name
- Master of Applied Science - MASc
- Level thesis:degree_level
- master's
- Discipline thesis:degree_discipline
- Electrical and Computer Engineering
- Grantor dc:publisher
- University of British Columbia
- Year dc:date
- 2024
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Ghaderi, Rouzbeh
- Contributors dc:contributor
-
- Thrampoulidis, Christos
Rights
dc:rights- Statement dc:rights
-
- Attribution-NonCommercial-NoDerivatives 4.0 International
- Language dc:language
- eng
Identifiers
dc:identifier.*- Handle dc:identifier
- http://hdl.handle.net/2429/89486