Back to search

University of British Columbia

Insights from infinitely wide neural networks

Abstract

dc:description

Studying neural networks in the limit of infinite-width has provided us with numerous valuable theoretical and practical insights about the initialization of NNs, their training dynamics and properties of the learnt functions. One of the theoretical tools emerged from this study is the empirical Neural Tangent Kernel (eNTK). The eNTK can provide a good understanding of a given network’s representation: they are often far less expensive to compute and applicable more broadly than infinite-width NTKs. In this work, we use eNTKs to predict the local dynamics of neural networks in an active learning setup to propose a new method for approximating active learning acquisition strategies that are based on retraining with hypothetically-labeled candidate data points. Furthermore, to tackle the notorious space and computational complexity of calculating eNTKs, we propose a fast approximation of eNTK using a block-diagonal kernel resulting from eNTK with respect to only one (or average) of the output neurons of a network. We further use this approximation in our proposed “look-ahead” strategies in deep active learning. We finally present empirical evidence that our querying strategy beats other look-ahead strategies by large margins, and achieves equal or better performance compared to state-of-the-art methods on several benchmark datasets in pool-based active learning.

Degree

thesis:*
Name thesis:degree_name
Master of Science - MSc
Level thesis:degree_level
master's
Discipline thesis:degree_discipline
Computer Science
Grantor dc:publisher
University of British Columbia
Year dc:date
2023

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Mohamadi, Mohamad Amin
Contributors dc:contributor
  • Sutherland, Danica J.

Rights

dc:rights
Statement dc:rights
  • Attribution-NonCommercial-NoDerivatives 4.0 International
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/84542

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Mohamadi, Mohamad Amin. Insights from infinitely wide neural networks. master's thesis, University of British Columbia, 2023. http://hdl.handle.net/2429/84542