Back to results

National University of Singapore

REGULARIZATION ON MACHINE LEARNING

Abstract

dc:description.abstract

Deep neural networks have become a powerful tool for machine learning problems. However, overfitting frequently occurs. To achieve better generalization, many regularization methods were proposed to reduce overfitting. In this thesis, we propose a simple-yet-effective regularization method called Drop-Activation. At the training phase, we drop nonlinear activation functions randomly and set them to be identity functions. At the testing phase, a deterministic network with a new activation function is used and the new activation function is designed to average effect of the randomness of discarding activations. We theoretically deduce the implicit regularization terms of Drop-Activation and the effect of Drop-Activation can be considered as implicit parameter reduction. Also, our theoretical analysis verifies its capability to be used together with Batch Normalization (Ioffe and Szegedy 2015). We perform Drop-Activation on the benchmark datasets and show that the performance of popular networks can be improved generally by Drop-Activation.

Degree

thesis:*
Name thesis:degree_name
MASTER OF SCIENCE (RSH-FOS)
Year dc:date.issued
2019

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • LIANG SENWEI
Advisor dc:contributor.supervisor
  • Yang Haizhao

Subjects

dc:subject × 1

Rights

Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Citation
LIANG SENWEI (2019-12-18). REGULARIZATION ON MACHINE LEARNING. ScholarBank@NUS Repository.

Chain of custody

source
Harvested from
National University of Singapore
Base URL
scholarbank.nus.edu.sg/oai/request
Last updated
2026-11-20
Source record
OAI-PMH GetRecord
citation

LIANG SENWEI. REGULARIZATION ON MACHINE LEARNING. 2019. https://scholarbank.nus.edu.sg/handle/10635/166275