Back to search

University of Saskatchewan

Systematic Exploration of Power-Augmented Feedforward Layers in Transformer Networks

Abstract

dc:description.abstract

Multi-layer perceptrons (MLPs) represent the foundational architecture of deep learning neural networks, consisting of fully-connected layers organized in a hierarchical structure. While activation functions between layers introduce non-linearity to the overall network, the fundamental computation within each layer remains strictly linear, where each neuron computes a weighted sum of its inputs. This linear nature of individual layer operations is a defining characteristic that constrains the expressiveness of each layer to linear transformations, with the network’s overall representational capacity determined by its depth (number of layers) and width (number of neurons per layer). In this thesis, the efficiency of using power-augmented networks in transformer models will be investigated. Power-augmented networks utilize trainable powers on their connections, allowing powers to be any real numbers. This introduces a more expressive form of learning non-linearity in neural networks and enables the learning of complex curves using non-linear power-augmented connections potentially enabling capturing of intricate patterns more efficiently. While conventional networks are fundamentally limited to learning piecewise linear functions, power- augmented functions can capture complex curves with fewer resources than traditional approaches. A crit- ical challenge addressed in this work involved ensuring numerical stability, particularly when dealing with fractional powers and small input values. Through systematic experiments, this thesis delivers an optimized NVIDIA CUDA implementation for the power-augmented perceptron layer and novel solutions to training stability challenges inherent in networks with dynamic power parameters. The presented evaluation framework demonstrates that power-augmented networks can achieve optimal performance with significant trade-offs across measurable metrics such as parameter count, validation loss, training time, and inference time. Key findings show that power-augmented networks can achieve superior or comparable validation loss performance to deeper baseline models while requiring fewer parameters and offering reductions in computation time and latency.

Degree

thesis:*
Name thesis:degree_name
Master of Science (M.Sc.)
Level thesis:degree_level
Masters
Discipline thesis:degree_discipline
Computer Science
Department thesis:degree_department
Computer Science
Grantor
University of Saskatchewan
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Salehipour, Parsa
Advisor dc:contributor.advisor
  • Jamali, Nadeem
Committee members dc:contributor.committeemember
  • Jamali, Nadeem
  • Rochan, Mrigank
  • Mondal, Debajyoti

Subjects

dc:subject × 1

Rights

Language dc:language.iso
en

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/10388/17550

Chain of custody

source
Harvested from
University of Saskatchewan
Base URL
harvest.usask.ca/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Salehipour, Parsa. Systematic Exploration of Power-Augmented Feedforward Layers in Transformer Networks. Masters thesis, University of Saskatchewan, 2025. https://hdl.handle.net/10388/17550