Back to search

Cornell University

On the Efficiency in Neural Networks: Exposing Unnecessary Computation

Abstract

dc:description.abstract

In the past decade, we have witnessed rapid progress in machine learning with the help of neural network models. Recently, neural models have achieved superhuman performance on various tasks such as image recognition, graph classification, and reading comprehension. With tremendous computation, researchers and engineers are able to train unprecedentedly large models and achieve superhuman performance on several benchmarks. While the performance (or accuracy) of neural models receives a significant amount of attention, there is an increasing concern about the efficiency and the environmental impact of these models since training a state-of-the-art model can be both costly and time-consuming. Moreover, while being accurate in offline settings, state-of-the-art models are seldom used in real-life applications due to their high latency and cost. To improve the performance of neural networks, researchers introduce more complex operations and models, which allow them to capture more complicated functions. While achieving higher accuracy, the reasons behind the success of several modern neural components, including graph convolution and self-attention remain concealed. Does the improvement come from additional parameters and model capacity or the induction bias? Do they utilize additional information? Besides new operations, stacking layers to increase the depth of a neural network is a common practice to enlarge the model capacity and improve the performance. While examples near the decision boundaries are hard to be classified, a shallow network with little computation can already deal with a large number of tasks. How can we dynamically allocate different computing resources to different examples at inference time? We will address these questions. This dissertation aims to expose unnecessary computation in modern neural networks and provide insights for designing efficient neural architectures. In chapters 3 and 4, we analyze two popular neural models, the graph convolution networks (GCNs) and the Transformers with self-attention mechanism, and propose simpler yet more efficient alternatives which achieve better or comparable performance. We point out that the non-linearity is unnecessary in GCNs and their success comes from feature propagation among local neighbors. We analyze the success of the Transformer models and demonstrate that convolution layers with particular emphasis on the local features can be very competitive as well. In chapters 5 and 6, we introduce novel neural architectures tailored to conditional inference strategies. On image recognition and machine reading comprehension, models with conditional inference strategies can achieve several times speedup through saving unnecessary computation on easy examples.

Degree

thesis:*
Name thesis:degree_name
Ph. D., Computer Science
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Computer Science
Grantor
Cornell University
Year dc:date.issued
2020

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Wu, Felix
Chair dc:contributor.chair
  • Weinberger, Kilian Quirin
Committee members dc:contributor.committeemember
  • Danescu-Niculescu-Mizil, Cristian
  • Sridharan, Karthik

Subjects

dc:subject × 5

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Bibid
13312078
Dc Identifier Other
ProQuest Submission ID: 12123
ProQuest Publication ID: 28029204

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Wu, Felix. On the Efficiency in Neural Networks: Exposing Unnecessary Computation. Doctor of Philosophy thesis, Cornell University, 2020. https://hdl.handle.net/1813/103432