Recurrent Neural Networks (RNNs) are variants of Neural Networks that are able to learn temporal relationships between sequences presented to the neural network. RNNs are often employed to learn underlying relationships in time series and sequential data.
This dissertation examines the extent of RNN’s memory retention and how it is influenced by different activation functions, network structures and recurrent network types.
To investigate memory retention, three approaches (and variants thereof) are used. First the number of patterns each network is able to retain is measured. Thereafter the length of retention is investigated. Lastly the previous experiments are combined to measure the retention of patterns over time. During each investigation, the effect of using different activation functions and network structures are considered to determine the configurations’ effect on memory retention.
The dissertation concludes that memory retention of a network is not necessarily improved when adding more parameters to a network. Activation functions have a large effect on the performance of RNNs when retaining patterns, especially temporal patterns.
Deeper network structures have the trade-off of less memory retention per parameter in favour of the ability to model more complex relationships. / Dissertation (MSc (Computer Science))--University of Pretoria, 2020. / Computer Science / MSc (Computer Science) / Unrestricted
| Identifer | oai:union.ndltd.org:netd.ac.za/oai:union.ndltd.org:up/oai:repository.up.ac.za:2263/78061 |
| Date | January 2020 |
| Creators | Pretorius, Abraham Daniel |
| Contributors | Cleghorn, Christopher W., u12022404@tuks.co.za |
| Publisher | University of Pretoria |
| Source Sets | South African National ETD Portal |
| Language | English |
| Detected Language | English |
| Type | Dissertation |
| Rights | © 2019 University of Pretoria. All rights reserved. The copyright in this work vests in the University of Pretoria. No part of this work may be reproduced or transmitted in any form or by any means, without the prior written permission of the University of Pretoria. |
Page generated in 0.0018 seconds