The optimisation and adaptation of single hidden layer feed-forward neural networks employing radial basis activation functions (RBFNs) was investigated. Previous work on RBFNs has mainly focused on problems with large data sets. The training algorithms developed with large data sets prove unreliable for problems with a small number of observations, a situation frequently encountered in process engineering. The primary objective of this study was the development of efficient and reliable learning algorithms for the training of RJBFNs with small and noisy data sets. It was demonstrated that regularisation is essential in order to filter out the noise and prevent over-fitting. The selection of the appropriate level of regularisation, lambda*, with small data sets presents a major challenge. The leave-one-out cross validation technique was considered as a potential means for automatic selection of lambda*. The computational burden of selecting lambda* was significantly reduced by a novel application of the generalised singular value decomposition. The exact solution of the multivariate linear regularisation problem can be represented as a single hidden layer neural network, the Regularisation Network, with one neurone for each distinct exemplar. A new formula was developed for automatic selection of the regularisation level for a Regularisation Network with given non-linearities. It was shown that the performance of a Regularisation Network is critically dependent on the non-linear parameters of the activation function employed; a point which has received surprisingly little attention. It was demonstrated that a measure of the effective degrees of freedom df(lambda*,alpha) of a Regularisation Network can be used to select the appropriate width of the local radial basis functions, alpha, based on the data alone. The one-to-one correspondence between the number of exemplars and the number of hidden neurones of a Regularisation Network may prove computationally prohibitive. The remedy is to use a network with a smaller number of neurones, the Generalised Radial Basis Function Network (GRBFN). The training of a GRBFN ultimately settles down to a large-scale non-linear optimisation problem. A novel sequential back-fit algorithm was developed for training the GRBFNs, which enabled the optimisation to proceed one neurone at a time. The new algorithm was tested with very promising results and its application to a simple chemical engineering process was demonstrated In some applications the overall response is composed of sharp localised features superimposed on a gently varying global background. Existing multivariate regression techniques as well as conventional neural networks are aimed at filtering the noise and recovering the overall response. An initial attempt was made at developing an Adaptive GRBFN to separate the local and global features. An efficient algorithm was developed simply by insisting that all the activation functions which are responsible for capturing the global trend should lie in the null space of the differential operator generating the activation function of the kernel based neurones. It was demonstrated that the proposed algorithm performs extremely well in the absence of strong global input interactions.

No description available.

Background:- Artificial Neural networks are motivated from biological nervous system and can be used for classification and forecasting the data. Each neural node contains activation function could be used for solving non-linear problems and optimization function to minimize the loss and give more accurate results. Neural networks are bustling in the field of machine learning, which inspired this study to analyse the performance variation based on the use of different combinations of the activation functions and optimization algorithms in terms of accuracy results and metrics recall and impact of data-set features on the performance of the neural networks. Objectives:- This study deals with an experiment to analyse the performance of the combinations are performing well and giving more results and to see impact of the feature segregation from data-set on the neural networks model performance. Methods:- The process involve the gathering of the data-sets, activation functions and optimization algorithm. Execute the network model using 7X5 different combinations of activation functions and optimization algorithm and analyse the performance of the neural networks. These models are tested upon the same data-set with some of the discarded features to know the effect on the performance of the neural networks. Results:- All the metrics for evaluating the neural networks presented in separate table and graphs are used to show growth and fall down of the activation function when associating with different optimization function. Impact of the individual feature on the performance of the neural network is also represented. Conclusions:- Out of 35 combinations, combinations made from optimizations algorithms Adam,RMSprop and Adagrad and activation functions ReLU,Softplus,Tanh Sigmoid and Hard_Sigmoid are selected based on the performance evaluation and data has impact on the performance of the combinations of the algorithms and activation functions which is also evaluated based on the experimentation. Individual features have their corresponding effect on the neural network.

Yes / This study introduces a quantum-inspired computational paradigm to address the unresolved problem of Convolutional Neural Networks (CNNs) using the Rectified Linear Unit (ReLU) activation function (AF), i.e., the ‘dying ReLU’. This problem impacts the accuracy and the reliability in image classification tasks for critical applications, such as in healthcare. The proposed approach builds on the classical ReLU and Leaky ReLU, applying the quantum principles of entanglement and superposition at a computational level to derive two novel AFs, respectively the ‘Quantum ReLU’ (QReLU) and the ‘modified-QReLU’ (m-QReLU). The proposed AFs were validated when coupled with a CNN using seven image datasets on classification tasks involving the detection of COVID-19 and Parkinson’s Disease (PD). The out-of-sample/test classification accuracy and reliability (precision, recall and F1-score) of the CNN were compared against those of the same classifier when using nine classical AFs, including ReLU-based variations. Findings indicate higher accuracy and reliability for the CNN when using either QReLU or m-QReLU on five of the seven datasets evaluated. Whilst retaining the best classification accuracy and reliability for handwritten digits recognition on the MNIST dataset (ACC = 99%, F1-score = 99%), avoiding the ‘dying ReLU’ problem via the proposed quantum AFs improved recognition of PD-related patterns from spiral drawings with the QReLU especially, which achieved the highest classification accuracy and reliability (ACC = 92%, F1-score = 93%). Therefore, with these increased accuracy and reliability, QReLU and m-QReLU can aid critical image classification tasks, such as diagnoses of COVID-19 and PD. / The authors declare that this was the result of a HEIF 2020 University of Bradford COVID-19 response-funded project ‘Quantum ReLU-based COVID-19 Detector: A Quantum Activation Function for Deep Learning to Improve Diagnostics and Prognostics of COVID-19 from Non-ionising Medical Imaging’. However, the funding source was not involved in conducting the study and/or preparing the article.

This project aims to classify different units on a wireless network with the use of their frequency response. This is in purpose to increase security when communicating over WiFi. We use a convolution neural network for finding symmetries in the frequency responses recorded from two different units. We used two pre-recorded sets of data which contained the same units but from two different locations. The project achieve an accuracy of 99.987%, with a 5 hidden layers CNN, when training and testing on one dataset. When training the neural network on one set and testing it on a second set, we achieve results below 54.12% for identifying the units. At the end we conclude that the amount of data needed, for achieving high enough accuracy, is to large for this method to be a practical solution for non-stationary units.

The far-reaching successes of deep neural networks in a wide variety of learning tasks have prompted research on how model properties account for high network performance. For a specific class of models whose activation functions are piecewise linear, one such property of interest is the number of linear regions that the network generates. Such models themselves define piecewise linear functions by partitioning input space into disjoint regions and fitting a different linear function on each such piece. It would be expected that the number or configuration of such regions would describe the model’s ability to fit complicated functions. However, previous works have shown difficulty in identifying linear regions as satisfactory predictors of model success. In this thesis, the question of whether the generation of linear regions due to training encode the properties of the learning problem is explored. More specifically, it is investigated whether change in linear region density due to model fitting is related to the geometric properties of the training data. In this work, data geometry is characterized in terms of the curvature of the underlying manifold. Models with ReLU activation functions are trained on a variety of regression problems defined on artificial manifolds and the change in linear region density is recorded along trajectories in input space. Learning is performed on problems defined on curves, surfaces and for image data. Experiments are repeated as the data geometry is varied and the change in density is compared with the manifold curvature measure used. In no experimental setting, was the observed change in density found to be clearly linked with curvature. However, density was observed to increase at points of discontinuity. This suggests that linear regions can in some instances model data complexities, however, the findings presented here do not support that data curvature is encoded by the formation of linear regions. Thus, the role that linear regions play in controlling the capacity of piecewise linear networks remains open. Future research is needed to gain further insights into how data geometry and linear regions are connected. / De breda framgångar som djupa neurala nätverk har uppvisat i en mängd olika inlärningsproblem har inspirerat ny forskning med syfte att förklara vilka modellegenskaper som resulterar i högpresterande nätverk. För neurala nätverk som använder styckvis linjära aktiveringsfunktioner är en intressant egenskap att studera de linjära regioner som nätverket genererar i det vektorrum som utgör träningsdatans definitionsmängd. Nätverk med styckvis linjära aktiveringsfunktioner delar upp definitionsmängden i distinkta regioner på vilka olika linjära funktioner avbildas. Dessa nätverk avbildar själva styckvis linjära funktioner. Genom att anpassa flera skilda linjära avbildningar går det att approximera funktioner som är icke-linjära. Därför skulle man kunna förvänta sig att antalet linjära regioner som en modell genererar och hur de är fördelade i rummet kunde fungera som mått på modellens förmåga att lära sig komplicerade funktioner. Tidigare efterforskingar inom detta område har dock inte kunnat demonstrera ett samband mellan antalet eller fördelningen av linjära regioner och modellens prestanda. I den här avhandlingen undersöks det vilken roll linjära regioner spelar i att förklara en modells kapacitet och vad den lär sig. Fångar de linjära regioner som ett nätverk lär sig de underliggande egenskaperna hos träningsdatan? Mer specifikt så studeras huruvida den lokala förändringen i antalet linjära regioner efter modellträning korrelerar med träningsdatans geometri. Träningsdata genereras från syntetiska mångfalder och datageometrin beskrivs i termer av mångfaldens krökning. På dessa mångfalder definieras regressionsproblem och träning upprepas för topologier av olika form och med olika krökning. Skillnaden i antalet linjära regioner efter träning mäts längs banor i definitionsdomänen och jämförs med datans krökning. Ingen av de experiment som utfördes lyckades påvisa något tydligt samband mellan förändring i antal regioner och datans krökning. Det observerades dock att antalet linjära regioner ökar i närheten av punkter som utgör diskontinuiteter. Detta antyder att linjära regioner under vissa omständigheter kan modellera komplexitet. Således förblir rollen som linjära regioner har i att förklara modellförmåga diffus.

Σε αυτή την εργασία, μελετάμε την κλάση των Υψηλής Τάξης Νευρωνικών Δικτύων και ειδικότερα των Πι—Σίγμα Νευρωνικών Δικτύων. Η απόδοση των Πι—Σίγμα Νευρωνικών Δικτύων αξιολογείται με την εφαρμογή τους σε διάφορα πολύ γνωστά χαρακτηριστικά προβλήματα εκπαίδευσης νευρωνικών δικτύων. Στα πειράματα που πραγματοποιήθηκαν, για την εκπαίδευση των Πι—Σίγμα Νευρωνικών Δικτύων υλοποιήθηκαν και εφαρμόστηκαν Σειριακοί και Παράλληλοι/Κατανεμημένοι Εξελικτικοί Αλγόριθμοι. Πιο συγκεκριμένα χρησιμοποιήθηκαν οι σειριακές καθώς και οι παράλληλες/κατανεμημένες εκδοχές των Διαφοροεξελικτικών Αλγόριθμων. Η προτεινόμενη μεθοδολογία βασίστηκε σε αυτές τις εκδοχές και εφαρμόστηκε για την εκπαίδευση των Πι—Σίγμα δικτύων χρησιμοποιώντας συναρτήσεις ενεργοποίησης «κατώφλια». Επιπρόσθετα, όλα τα βάρη και οι μεροληψίες των δικτύων περιορίστηκαν σε ένα μικρό εύρος ακέραιων αριθμών, στο διάστημα [-32, 32]. Συνεπώς, τα εκπαιδευμένα Πι—Σίγμα νευρωνικά δίκτυα μπορούν να αναπαρασταθούν με ακεραίους των 6-bits. Αυτής της μορφής τα δίκτυα είναι πιο κατάλληλα για την εφαρμογή τους σε «υλικό» (hardware), από νευρωνικά δίκτυα με πραγματικά βάρη. Τα πειραματικά αποτελέσματα μας δείχνουν ότι η διαδικασία εκπαίδευσης είναι γρήγορη, σταθερή και αξιόπιστη. Ακόμα η εφαρμογή των παράλληλων/κατανεμημένων Εξελικτικών Αλγορίθμων για την εκπαίδευση των Πι—Σίγμα δικτύων μας επιδεικνύει αρκετά καλές ικανότητες γενίκευσης των εκπαιδευμένων δικτύων καθώς και προσφέρει επιτάχυνση στην διαδικασία εκπαίδευσης τους. / In this contribution, we study the class of Higher-Order Neural Networks and especially the Pi-Sigma Networks. The performance of Pi-Sigma Networks is evaluated through several well known neural network training benchmarks. In the experiments reported here, Evolutionary Algorithms and Parallel/Distributed Evolutionary Algorithms are implemented for Pi-Sigma neural networks training. More specifically the serial as well as a parallel/distributed version of the Differential Evolution have been employed. The proposed approach is applied to train Pi-Sigma networks using threshold activation functions. Moreover, the weights and biases were confined to a narrow band of integers, constrained in the range [-32, 32]. Thus the trained Pi-Sigma neural networks can be represented by just 6 bits. Such networks are better suited for hardware implementation than the real weight ones. Experimental results suggest that this training process is fast, stable and reliable and the trained Pi-Sigma networks, with both serial and parallel/distributed algorithms, exhibited good generalization capabilities. Furthermore, the usage of a distributed version of the Differential Evolution, has demonstrated a speedup of the training process.

With the arrival of automated vehicles (AVs) on our streets virtually around the corner, this thesis explores advances in automated driving technology with a focus on ethical decision making in dilemmatic traf- fic situations. In a total of five publications, we take a multi-facetted approach to analyse and address the core challenges related to auto- mated ethical decision making in AVs. In publications one through three, we conduct a series of immersive virtual reality studies to analyze human behavior in traffic dilemmas, explore mathematical approaches to model the decision making process, investigate how the assessment methodology can affect moral judgment, and discuss the implications of these studies for algorithmic decision making in the real-world. In publication number four, we provide a comprehensive summary of the status quo of AV technology and legislation with regard to automated ethical decision making. Here, we discuss when and why ethical deci- sion making systems become necessary in AVs, review existing guide- lines for the behavior of AVs in dilemma situations, and compile a set of 10 demands and open questions that need to be addressed in the pursuit of a framework for ethical decision making in AVs. Finally, the basis for automated ethical decision making in AVs will be provided by accurate assessments of the immediate environment of the car. The pri- mary technology used to provide the required information processing of camera and LiDAR images in AVs is machine learning, and in particular deep learning. In publication five, we propose a form of adaptive acti- vation functions, addressing a central element of deep neural networks, which could, for instance, lead to increased detection rates of relevant objects, and thus help to provide a more accurate assessment of the AVs environment. Overall, this thesis provides a structured and compre- hensive overview of the state of the art in ethical decision making for AVs. It includes important implications for the design of decision mak- ing algorithms in practice, and concisely outlines the central remaining challenges on the road to a safe, fair and successful introduction of fully automated vehicles into the market.