The generalization ability of artificial neural networks (ANN) is highly dependent on their architectures and can be critical to solving a given problem. The current best practice uses fixed architectures determined via a trial-and-error approach. This process can be both computationally and temporally cumbersome and does not guarantee that an optimal topology will even be found. Replacing the user’s role in designing topologies with methods that enable a system to manage its own growth can endow systems with adaptable learning.
Constructive algorithms offer the possibility of compact architectures as an alternative to the trial-and-error approach. This class of algorithms grows a network’s topology by incrementally adding units during learning to match task complexity. However, the decision of when to add new units in constructive algorithms heavily depends on user-defined a priori hyperparameters, which can be task-specific. Contrary to having a user fine-tune hyperparameters that govern growth, the intrinsic population dynamics of an ANN could be used to self-govern the growing process. Theoretically, an ANN or each layer comprising the network can be viewed as a set of populations. From this perspective, a hidden layer can be considered the environment in which hidden units exist. In this work, we propose a novel, more self-governed growing algorithm inspired by population dynamics for determining near-optimal topologies of feedforward ANNs. This allows the inclusion of a carrying capacity, the maximum population of hidden units that can be sustained in a hidden layer. Including this constraint in combination with population dynamics provides a built-in mechanism for a dynamic growth rate. The proposed approach is used in parallel with direct performance feedback from the network to modulate the growth rate of the hidden layer, allowing the network to converge to smaller topologies based on the task's demands. More self-governed approaches reduce the number of finely-tuned hyperparameters required to decide when to grow and put more control of the network’s structure and representational capacities in the algorithms themselves, facilitating the emergence of inherent intelligent behaviour.
Chapter one introduces a dynamic, more self-governed growing algorithm inspired by population dynamics. Results show that compared to using fixed rules for determining hidden layer sizes; dynamic growth leads to smaller topologies than predicted while still being capable of solving the task. In chapter two, we investigate the algorithm's inherent properties to validate the more self-governed aspect. The results depict that the model’s hyperparameters require less fine-tuning by the user and adhere more toward self-governance. Finally, in chapter three, we investigate the effects of growing hidden layers individually in a sequential fashion or simultaneously in a parallel fashion multilayer context. A modified version of the growing algorithm capable of growing parallel is proposed. Growing hidden layers in parallel resulted in comparable or higher performances than sequential approaches. The growing algorithm presented here offers more self-governed growth, which provides an effective general solution automatically tailored to the task.
| Identifer | oai:union.ndltd.org:uottawa.ca/oai:ruor.uottawa.ca:10393/45674 |
| Date | 28 November 2023 |
| Creators | Ross, Matthew |
| Contributors | Chartier, Sylvain J. |
| Publisher | Université d'Ottawa / University of Ottawa |
| Source Sets | Université d’Ottawa |
| Language | English |
| Detected Language | English |
| Type | Thesis |
| Format | application/pdf |
| Rights | Attribution-NonCommercial-NoDerivatives 4.0 International, http://creativecommons.org/licenses/by-nc-nd/4.0/ |
Page generated in 0.0023 seconds