M. Ing. / This dissertation describes the development of a feedforwa.rd neural network, trained by means of an accelerated backpropagation algorithm, used for the short term load forecasting on real world data. It is argued that the new learning algorithm. I-Prop, - is a faster training - algorithm due to the fact that the learning rate is optimally predicted and changed according to a more efficient formula (without the need for extensive memory) which speeds up the training process. The neural network developed was tested for the month of December 1994, specifically to test the artificial neural network's ability to correctly predict the load during a Public Holiday, as well as the change over from Public Holiday to 'Normal' working day. In conclusion, suggestions are made towards further research in the improvement of the I-Prop algorithm as well as improving the load forecasting technique implemented in this dissertation.

Spread F is a phenomenon of the ionosphere in which the pulses returned from the ionosphere are of a much greater duration than the transmitted ones. The occurrence of spread F can be predicted using the technique of Neural Networks (NNs). This thesis presents the development and evaluation of NN based models (two single station models and a regional model) for predicting the occurrence of spread F over selected stations within the Brazilian sector. The input space for the NNs included the day number (seasonal variation), hour (diurnal variation), sunspot number (measure of the solar activity), magnetic index (measure of the magnetic activity) and magnetic position (latitude, magnetic declination and inclination). Twelve years of spread F data measured during 1978 to 1989 inclusively at the equatorial site Fortaleza and low latitude site Cachoeira Paulista are used in the development of an input space and NN architecture for the NN models. Spread F data that is believed to be related to plasma bubble developments (range spread F) were used in the development of the models while those associated with narrow spectrum irregularities that occur near the F layer (frequency spread F) were excluded. The results of the models show the dependency of the probability of spread F as a function of local time, season and latitude. The models also illustrate some characteristics of spread F such as the onset and peak occurrence of spread F as a function of distance from the equator. Results from these models are presented in this thesis and compared to measured data and to modelled data obtained with an empirical model developed for the same purpose.

This thesis describes the search for a temporal model for predicting the peak ionospheric electron density-(foF2). Existing models, such as the International Reference Ionosphere (IRI) and 8KYCOM, were used to predict the 12 noon foF2 value over Grahamstown (26°E, 33°8). An attempt was then made to find a model that would improve upon these results. The traditional method of linear regression was used as a first step towards a new model. It was found that this would involve a multi variable regression that is reliant on guessing the optimum variables to be used in the final equation. An extremely complicated modelling equation involving many terms would result. Neural networks (NNs) are introduced as a new technique for predicting foF2. They are also applied, for the first time, to the problem of determining the best predictors of foF2. This quantity depends upon day number, level of solar activity and level of magnetic activity. The optimum averaging lengths of the solar activity index and the magnetic activity index were determined by appling NNs, using the criterion that the best indices are those that give the lowest rms error between the measured and predicted foF2. The optimum index for solar activity was found to be a 2-month running mean value of the daily sunspot number and for magnetic activity a 2-day averaged A index was found to be optimum. In addition, it was found that the response of foF2 to magnetic activity changes is highly non-linear and seasonally dependent. Using these indices as inputs, the NN trained successfully to predict foF2 with an rms error of 0.946 MHz on the daily testing values. Comparison with the IRI showed an improvement of 40% on the rms error. It is also shown that the NN will predict the noon value of foF2 to the same level of accuracy for unseen data of the same type.

M.Ing. (Electrical And Electronic Engineering) / This dissertation describes the development of a system of Artificial Neural Networks that enables the incremental training of feed forward neural networks using supervised training algorithms such as back propagation. It is argued that incremental learning is fundamental to the adaptive learning behavior observed in human intelligence and constitutes an imperative step towards artificial cognition. The importance of developing incremental learning as a system of ANNs is stressed before the complete system is presented. Details of the development and implementation of the system is complemented by the description of two case studies. In conclusion the role of the incremental learning system as basis for further development of fundamental elements of cognition is projected.

M.Ing. (Mechanical Engineering) / Please refer to full text to view abstract

M.Ing. (Mechanical Engineering) / This thesis involves the development of a Neural Network software package within a Windows environment. This package is called NetPro. It contains most of the standard tools used in existing neural network packages e.g. shuffling of facts, automatic test file facts extraction, randomizing of weights values (before and during training), automatic/manual construction of network files, logging of network properties during training, noise can be added to inputs, etc. NetPro has three additional tools: (a) time delay actions on inputs, (b) a neural network calculator, and (c) automatic saving of the best network during training. The calculator is used to calculate the number of training facts needed for optimum generalization ...

In the domain of strategic game playing, the use of statistical techniques such as the Upper Confidence for Trees (UCT) algorithm, has become the norm as they offer many benefits over classical algorithms. These benefits include requiring no game-specific strategic knowledge and time-scalable performance. UCT does not incorporate any strategic information specific to the game considered, but instead uses repeated sampling to effectively brute-force search through the game tree or search space. The lack of game-specific knowledge in UCT is thus both a benefit but also a strategic disadvantage. Pattern recognition techniques, specifically Neural Networks (NN), were identified as a means of addressing the lack of game-specific knowledge in UCT. Through a novel hybridisation technique which combines UCT and trained NNs for pruning, the UCTNN algorithm was derived. The NN component of UCT-NN was trained using a UCT self-play scheme to generate game-specific knowledge without the need to construct and manage game databases for training purposes. The UCT-NN algorithm is outlined for pruning in the game of Go-Moku as a candidate case-study for this research. The UCT-NN algorithm contained three major parameters which emerged from the UCT algorithm, the use of NNs and the pruning schemes considered. Suitable methods for finding candidate values for these three parameters were outlined and applied to the game of Go-Moku on a 5 by 5 board. An empirical investigation of the playing performance of UCT-NN was conducted in comparison to UCT through three benchmarks. The benchmarks comprise a common randomly moving opponent, a common UCTmax player which is given a large amount of playing time, and a pair-wise tournament between UCT-NN and UCT. The results of the performance evaluation for 5 by 5 Go-Moku were promising, which prompted an evaluation of a larger 9 by 9 Go-Moku board. The results of both evaluations indicate that the time allocated to the UCT-NN algorithm directly affects its performance when compared to UCT. The UCT-NN algorithm generally performs better than UCT in games with very limited time-constraints in all benchmarks considered except when playing against a randomly moving player in 9 by 9 Go-Moku. In real-time and near-real-time Go-Moku games, UCT-NN provides statistically significant improvements compared to UCT. The findings of this research contribute to the realisation of applying game-specific knowledge to the UCT algorithm.

Modern neural network (NN) based control schemes have surmounted many of the limitations found in the traditional control approaches. Nevertheless, these modern control techniques have only recently been introduced for use on high-specification Programmable Logic Controllers (PLCs) and usually at a very high cost in terms of the required software and hardware. This ‗intelligent‘ control in the sector of industrial automation, specifically on standard PLCs thus remains an area of study that is open to further research and development. The research documented in this thesis examined the effectiveness of linear traditional control schemes such as Proportional Integral Derivative (PID), Lead and Lead-Lag control, in comparison to non-linear NN based control schemes when applied on a strongly non-linear platform. To this end, a mechatronic-type balancing system, namely, the Ball-on-Wheel (BOW) system was designed, constructed and modelled. Thereafter various traditional and intelligent controllers were implemented in order to control the system. The BOW platform may be taken to represent any single-input, single-output (SISO) non-linear system in use in the real world. The system makes use of current industrial technology including a standard PLC as the digital computational platform, a servo drive and wireless access for remote control. The results gathered from the research revealed that NN based control schemes (i.e. Pure NN and NN-PID), although comparatively slower in response, have greater advantages over traditional controllers in that they are able to adapt to external system changes as well as system non-linearity through a process of learning. These controllers also reduce the guess work that is usually involved with the traditional control approaches where cumbersome modelling, linearization or manual tuning is required. Furthermore, the research showed that online-learning adaptive traditional controllers such as the NN-PID controller which maintains the best of both the intelligent and traditional controllers may be implemented easily and with minimum expense on standard PLCs.

Despite the impressive performance of deep neural networks (DNNs) on numerous vision tasks, they still exhibit yet-to-understand uncouth behaviours. One puzzling behaviour is the reaction of DNNs to various noise attacks, where it has been shown that there exist small adversarial noise that can result in a severe degradation in the performance of DNNs. To rigorously treat this, we derive exact analytic expressions for the first and second moments (mean and variance) of a small piecewise linear (PL) network with a single rectified linear unit (ReLU) layer subject to general Gaussian input. We experimentally show that these expressions are tight under simple linearizations of deeper PL-DNNs, especially popular architectures in the literature (e.g. LeNet and AlexNet). Extensive experiments on image classification show that these expressions can be used to study the behaviour of the output mean of the logits for each class, the inter-class confusion and the pixel-level spatial noise sensitivity of the network. Moreover, we show how these expressions can be used to systematically construct targeted and non-targeted adversarial attacks. Then, we proposed a special estimator DNN, named mixture of linearizations (MoL), and derived the analytic expressions for its output mean and variance, as well. We employed these expressions to train the model to be particularly robust against Gaussian attacks without the need for data augmentation. Upon training this network on a loss that is consolidated with the derived output probabilistic moments, the network is not only robust under very high variance Gaussian attacks but is also as robust as networks that are trained with 20 fold data augmentation.

A Thesis submitted to the Faculty of Science, University of the Witwatersrand, Johannesburg, in fulfilment of the requirements for the degree of Doctor of Philosophy, 2017 / The utility of lectures is directly related to the engagement of the students therein. To ensure the value of lectures, one needs to be certain that they are engaging to students. In small classes experienced lecturers develop an intuition of how engaged the class is as a whole and can then react appropriately to remedy the situation through various strategies such as breaks or changes in style, pace and content. As both the number of students and size of the venue grow, this type of contingent teaching becomes increasingly difficult and less precise. Furthermore, relying on intuition alone gives no way to recall and analyse previous classes or to objectively investigate trends over time. To address these problems this thesis presents the WITS INTELLIGENT TEACHING SYSTEM (WITS) to highlight disengaged students during class. A web-based, mobile application called Engage was developed to try elicit anonymous engagement information directly from students. The majority of students were unwilling or unable to self-report their engagement levels during class. This stems from a number of cultural and practical issues related to social display rules, unreliable internet connections, data costs, and distractions. This result highlights the need for a non-intrusive system that does not require the active participation of students. A nonintrusive, computer vision and machine learning based approach is therefore proposed. To support the development thereof, a labelled video dataset of students was built by recording a number of first year lectures. Students were labelled across a number of affects – including boredom, frustration, confusion, and fatigue – but poor inter-rater reliability meant that these labels could not be used as ground truth. Based on manual coding methods identified in the literature, a number of actions, gestures, and postures were identified as proxies of behavioural engagement. These proxies are then used in an observational checklist to mark students as engaged or not. A Support Vector Machine (SVM) was trained on Histograms of Oriented Gradients (HOG) to classify the students based on the identified behaviours. The results suggest a high temporal correlation of a single subject’s video frames. This leads to extremely high accuracies on seen subjects. However, this approach generalised poorly to unseen subjects and more careful feature engineering is required. The use of Convolutional Neural Networks (CNNs) improved the classification accuracy substantially, both over a single subject and when generalising to unseen subjects. While more computationally expensive than the SVM, the CNN approach lends itself to parallelism using Graphics Processing Units (GPUs). With GPU hardware acceleration, the system is able to run in near real-time and with further optimisations a real-time classifier is feasible. The classifier provides engagement values, which can be displayed to the lecturer live during class. This information is displayed as an Interest Map which highlights spatial areas of disengagement. The lecturer can then make informed decisions about how to progress with the class, what teaching styles to employ, and on which students to focus. An Interest Map was presented to lecturers and professors at the University of the Witwatersrand yielding 131 responses. The vast majority of respondents indicated that they would like to receive live engagement feedback during class, that they found the Interest Map an intuitive visualisation tool, and that they would be interested in using such technology. Contributions of this thesis include the development of a labelled video dataset; the development of a web based system to allow students to self-report engagement; the development of cross-platform, open-source software for spatial, action and affect labelling; the application of Histogram of Oriented Gradient based Support Vector Machines, and Deep Convolutional Neural Networks to classify this data; the development of an Interest Map to intuitively display engagement information to presenters; and finally an analysis of acceptance of such a system by educators. / XL2017