A new scheme for training Rectified Linear Unit (ReLU) based feedforward neural networks is examined in this thesis. The project starts with the row-by-row updating strategy designed for Single-hidden Layer Feedforward neural Networks (SLFNs). This strategy exploits the properties held by ReLUs and optimizes each row in the input weight matrix individually, under the common optimization scheme. Then the Direct Updating Strategy (DUS), which has two different versions: Vector-Based Method (VBM) and Matrix-Based Method (MBM), is proposed to optimize the input weight matrix as a whole. Finally DUS is extended to Multi-hidden Layer Feedforward neural Networks (MLFNs). Since the extension, for general ReLU-based MLFNs, faces an initialization dilemma, a special structure MLFN is presented. Verification experiments are conducted on six benchmark multi-class classification datasets. The results confirm that MBM algorithm for SLFNs improves the performance of neural networks, compared to its competitor, regularized extreme learning machine. For most datasets involved, MLFNs with the proposed special structure perform better when adding extra hidden layers. / Ett nytt schema för träning av rektifierad linjär enhet (ReLU)-baserade och framkopplade neurala nätverk undersöks i denna avhandling. Projektet börjar med en rad-för-rad-uppdateringsstrategi designad för framkopplade neurala nätverk med ett dolt lager (SLFNs). Denna strategi utnyttjar egenskaper i ReLUs och optimerar varje rad i inmatningsviktmatrisen individuellt, enligt en gemensam optimeringsmetod. Därefter föreslås den direkta uppdateringsstrategin (DUS), som har två olika versioner: vektorbaserad metod (VBM) respektive matrisbaserad metod (MBM), för att optimera ingångsviktmatrisen som helhet. Slutli- gen utvidgas DUS till framkopplade neurala nätverk med flera lager (MLFN). Eftersom utvidgningen för generella ReLU-baserade MLFN står inför ett initieringsdilemma presenteras därför en MLFN med en speciell struktur. Verifieringsexperiment utförs på sex datamängder för klassificering av flera klasser. Resultaten bekräftar att MBM-algoritmen för SLFN förbättrar prestanda hos neurala nätverk, jämfört med konkurrenten, den regulariserade extrema inlärningsmaskinen. För de flesta använda dataset, fungerar MLFNs med den föreslagna speciella strukturen bättre när man lägger till extra dolda lager.

Various techniques of optimizing the multiple class cross-entropy error function to train single hidden layer neural network classifiers with softmax output transfer functions are investigated on a real-world multispectral pixel-by-pixel classification problem that is of fundamental importance in remote sensing. These techniques include epoch-based and batch versions of backpropagation of gradient descent, PR-conjugate gradient and BFGS quasi-Newton errors. The method of choice depends upon the nature of the learning task and whether one wants to optimize learning for speed or generalization performance. It was found that, comparatively considered, gradient descent error backpropagation provided the best and most stable out-of-sample performance results across batch and epoch-based modes of operation. If the goal is to maximize learning speed and a sacrifice in generalisation is acceptable, then PR-conjugate gradient error backpropagation tends to be superior. If the training set is very large, stochastic epoch-based versions of local optimizers should be chosen utilizing a larger rather than a smaller epoch size to avoid inacceptable instabilities in the generalization results. (authors' abstract) / Series: Discussion Papers of the Institute for Economic Geography and GIScience

The Littoral Acoustic Demonstration Center has collected passive acoustic monitoring data in the northern Gulf of Mexico since 2001. Recordings were made in 2007 near the Deepwater Horizon oil spill that provide a baseline for an extensive study of regional marine mammal populations in response to the disaster. Animal density estimates can be derived from detections of echolocation signals in the acoustic data. Beaked whales are of particular interest as they remain one of the least understood groups of marine mammals, and relatively few abundance estimates exist. Efficient methods for classifying detected echolocation transients are essential for mining long-term passive acoustic data. In this study, three data clustering routines using k-means, self-organizing maps, and spectral clustering were tested with various features of detected echolocation transients. Several methods effectively isolated the echolocation signals of regional beaked whales at the species level. Feedforward neural network classifiers were also evaluated, and performed with high accuracy under various noise conditions. The waveform fractal dimension was tested as a feature for marine biosonar classification and improved the accuracy of the classifiers. [This research was made possible by a grant from The Gulf of Mexico Research Initiative. Data are publicly available through the Gulf of Mexico Research Initiative Information & Data Cooperative (GRIIDC) at https://data.gulfresearchinitiative.org.] [DOIs: 10.7266/N7W094CG, 10.7266/N7QF8R9K]

Pagrindinis darbo tikslas – atlikti kelių tiesioginio sklidimo neuroninių tinklų sistemų lyginamąją analizę siekiant įvertinti jų funkcionalumą. Šiame darbe apžvelgiama: biologinio ir dirbtinio neuronų modeliai, neuroninių tinklų klasifikacija pagal jungimo konstrukciją (tiesioginio sklidimo ir rekurentiniai neuroniniai tinklai), dirbtinių neuroninių tinklų mokymo strategijos (mokymas su mokytoju, mokymas be mokytojo, hibridinis mokymas). Analizuojami pagrindiniai tiesioginio sklidimo neuroninių tinklų metodai: vienasluoksnis perceptronas, daugiasluoksnis perceptronas realizuotas „klaidos skleidimo atgal” algoritmu, radialinių bazinių funkcijų neuroninis tinklas. Buvo nagrinėjama 14 skirtingų tiesioginio sklidimo neuroninių tinklų sistemos. Programos buvo suklasifikuotos pagal kainą, tiesioginio sklidimo neuroninių tinklo mokymo metodų taikymą, galimybę vartotojui keisti parametrus prieš apmokant tinklą ir techninį programos įvertinimą. Programos buvo įvertintos dešimtbalėje vertinimo sistemoje pagal mokymo metodų įvairumą, parametrų keitimo galimybes, programos stabilumą, kokybę, bei kainos ir kokybės santykį. Aukščiausiu balu įvertinta „Matlab” programa (10 balų), o prasčiausiai – „Sharky NN” (2 balai). Detalesnei analizei pasirinktos keturios programos („Matlab“, „DTREG“, „PathFinder“, „Cortex“), kurios buvo įvertintos aukščiausiais balais, galėjo apmokyti tiesioginio sklidimo neuroninį tinklą daugiasluoksnio perceptrono metodu ir bent dvi radialinių bazinių funkcijų... [toliau žr. visą tekstą] / The main aim – to perform a comparative analysis of several feedforward neural system networks in order to identify its functionality. The work presents both: biological and artificial neural models, also classification of neural networks, according to connections’ construction (of feedforward and recurrent neural networks), studying strategies of artificial neural networks (with a trainer, without a trainer, hybrid). The main methods of feedforward neural networks: one-layer perceptron, multilayer perceptron, implemented upon “error feedback” algorithm, also a neural network of radial base functions have been considered. The work has included 14 different feedforward neural system networks, classified according its price, application of study methods of feedforward neural networks, also a customer’s possibility to change parameters before paying for the network and a technical evaluation of a program. The programs have been evaluated from 1 point to 10 points according to the following: variety of training systems, possibility to change parameters, stability, quality and ratio of price and quality. The highest evaluation has been awarded to “Matlab” (10 points), the lowest – to “Sharky NN” (2 points). Four programs (”Matlab“, “DTREG“, “PathFinder“,”Cortex“) have been selected for a detail analysis. The best evaluated programs have been able to train feedforward neural networks using multilayer perceptron method, also at least two radial base function networks. “Matlab“ and... [to full text]

The development of machine learning research has provided statistical innovations and further developments within the field of time series analysis. This study seeks to investigate two different approaches on artificial neural network models based on different learning techniques, and answering how well the neural network approach compares with a basic autoregressive approach, as well as how the artificial neural network models compare to each other. The models were compared and analyzed in regards to the univariate forecast accuracy on 20 randomly drawn time series from two different time frequencies from the M4 competition dataset. Forecasting was made dependent on one time lag (t-1) and forecasted three and six steps ahead respectively. The artificial neural network models outperformed the baseline Autoregressive model, showing notably lower mean average percentage error overall. The Multilayered perceptron models performed better than the Long short-term memory model overall, whereas the Long short-term memory model showed improvement on longer prediction time dimensions. As the training were done univariately  on a limited set of time steps, it is believed that the one layered-approach gave a good enough approximation on the data, whereas the added layer couldn’t fully utilize its strengths of processing power. Likewise, the Long short-term memory model couldn’t fully demonstrate the advantagements of recurrent learning. Using the same dataset, further studies could be made with another approach to data processing. Implementing an unsupervised approach of clustering the data before analysis, the same models could be tested with multivariate analysis on models trained on multiple time series simultaneously.

We consider a mechanistic non-linear machine learning approach to learning signals in financial time series data. A modularised and decoupled algorithm framework is established and is proven on daily sampled closing time-series data for JSE equity markets. The input patterns are based on input data vectors of data windows preprocessed into a sequence of daily, weekly and monthly or quarterly sampled feature measurement changes (log feature fluctuations). The data processing is split into a batch processed step where features are learnt using a Stacked AutoEncoder (SAE) via unsupervised learning, and then both batch and online supervised learning are carried out on Feedforward Neural Networks (FNNs) using these features. The FNN output is a point prediction of measured time-series feature fluctuations (log differenced data) in the future (ex-post). Weight initializations for these networks are implemented with restricted Boltzmann machine pretraining, and variance based initializations. The validity of the FNN backtest results are shown under a rigorous assessment of backtest overfitting using both Combinatorially Symmetrical Cross Validation and Probabilistic and Deflated Sharpe Ratios. Results are further used to develop a view on the phenomenology of financial markets and the value of complex historical data under unstable dynamics.

This thesis aims to solve the mathematical inverse problem of characterizing sea waves based on the responses obtained from a marine vessel sailing under certain sea conditions. By researching this problem the thesis contributes to the marine industry by improving products that are using ocean behavior for controlling ship's dynamics. Knowledge about the current state of the sea, such as the wave frequency and height, is important for navigation, control, and for the safety of a vessel. This information can be retrieved from specialized weather reports. However, such information is not at all time possible to obtain during a voyage, and if so usually comes with a certain delay. Therefore this thesis seeks solutions that can estimate on-line the waves' state using methods in the field of Artificial Intelligence. The specific investigation methods are Transfer Functions augmented with Genetic Algorithm, Artificial Neural Networks and Case-Based Reasoning. These methods have been configured and validated using the n-fold cross validation method. All the methods have been tested with an actual implementation. The algorithms have been trained with data acquired from a marine simulation program developed in Simulink. The methods have also been trained and tested using monitored data acquired from an actual ship sailing on the Baltic Sea as well as wave data obtained from a buoy located nearby the vessel's route. The proposed methods have been compared with state-of-the art reports in order evaluate the novelty of the research and its potential applications in industry. The results in this thesis show that the proposed methods can in fact be used for solving the inverse problem. It was also found that among the investigated methods it is the Transfer Function augmented with Genetic Algorithm which yields best results. This Master Thesis is conducted under the Master of Engineering Program in Robotics at Mälardalens högskola in Västerås, Sweden. The thesis was proposed by Q-TAGG R&D AB in Västerås, Sweden, a company which specializes in marine vessel dynamics research.

The growing urgency of climate change has led to growth in the electrification technology field, where batteries have emerged as an essential role in the renewable energy transition, supporting the implementation of environmentally friendly technologies such as smart grids, energy storage systems, and electric vehicles. Battery cell degradation is a common occurrence indicating battery usage. Optimizing lithium-ion battery degradation during operation benefits the prediction of future degradation, minimizing the degradation mechanisms that result in power fade and capacity fade. This degree project aims to investigate battery degradation prediction based on capacity using deep learning methods. Through analysis of battery degradation and health prediction for lithium-ion cells using non-destructive techniques. Such as electrochemical impedance spectroscopy obtaining ECM and three different deep learning models using multi-channel data. Additionally, the AI models were designed and developed using multi-channel data and evaluated performance within MATLAB. The results reveal an increased resistance from EIS measurements as an indicator of ongoing battery aging processes such as loss o active materials, solid-electrolyte interphase thickening, and lithium plating. The AI models demonstrate accurate capacity estimation, with the LSTM model revealing exceptional performance based on the model evaluation with RMSE. These findings highlight the importance of carefully managing battery charging processes and considering factors contributing to degradation. Understanding degradation mechanisms enables the development of strategies to mitigate aging processes and extend battery lifespan, ultimately leading to improved performance.