This research investigates developing a training method for Restricted Hopfield Network (RHN) which is a subcategory of Hopfield Networks. Hopfield Networks are recurrent neural networks proposed in 1982 by John Hopfield. They are useful for different applications such as pattern restoration, pattern completion/generalization, and pattern association. In this study, we propose an enhanced training method for RHN which not only improves the convergence of the training sub-routine, but also is shown to enhance the learning capability of the network. Particularly, after describing the architecture/components of the model, we propose a modified variant of SPSA which in conjunction with back-propagation over time result in a training algorithm with an enhanced convergence for RHN. The trained network is also shown to achieve a better memory recall in the presence of noisy/distorted input. We perform several experiments, using various datasets, to verify the convergence of the training sub-routine, evaluate the impact of different parameters of the model, and compare the performance of the trained RHN in recreating distorted input patterns compared to conventional RBM and Hopfield network and other training methods.

No description available.

The process of energy consumption and monitoring of a buildingis time-consuming. Therefore, an feasible approach for using trans-fer learning is presented to decrease the necessary time to extract re-quired large dataset. The technique applies a bidirectional long shortterm memory recurrent neural network using sequence to sequenceprediction. The idea involves a training phase that extracts informa-tion and patterns of a building that is presented with a reasonablysized dataset. The validation phase uses a dataset that is not sufficientin size. This dataset was acquired through a related paper, the resultscan therefore be validated accordingly. The conducted experimentsinclude four cases that involve different strategies in training and val-idation phases and percentages of fine-tuning. Our proposed modelgenerated better scores in terms of prediction performance comparedto the related paper.

No description available.

Supervised machine learning (ML) projects require data for model training, validation, and testing. However, the confidential nature of field and well production data often hinders the progress of ML projects. To address this issue, we developed a well simulator that generates realistic well production data based on physical, governing differential equations. The simulation models the reservoir, wellbore, flowline, and choke coupled using transient nodal analysis to solve for transient flow rate, pressure, and temperature as a function of variable choke opening over time in addition to a wide range of static parameters for each component. The simulator’s output is then perturbed using the gauge transfer function to introduce systematic and random errors, creating a dataset for ML projects without the need for confidential production data. We then generated a simulated dataset to train a recurrent neural network (RNN) on the task of classifying well on/off times. This task typically requires a significant number of manhours to manually filter and verify data for hundreds or thousands of wells. Our RNN model achieves high accuracy in classifying the correct on/off labels, representing a promising step towards a fully-automated rate allocation process. Our simulator for well production data can be used for other ML projects, circumventing the need for confidential data, and enabling the study and development of different ML models to streamline and automate various oil and gas work processes. Overall, the success of our RNN model demonstrates the potential of ML to improve the operational efficiency of various oil and gas work processes.

No description available.

Churn prediction is used to identify customers that are becoming less loyal and is an important tool for companies that want to stay competitive in a rapidly growing market. In retail, a dynamic definition of churn is needed to identify churners correctly. Customer Lifetime Value (CLV) is the monetary value of a customer relationship. No change in CLV for a given customer indicates a decrease in loyalty. This thesis proposes a novel approach to churn prediction. The proposed model uses a Recurrent Neural Network to identify churners based on Customer Lifetime Value time series regression. The results show that the model performs better than random. This thesis also investigated the use of the K-means algorithm as a replacement to a rule-extraction algorithm. The K-means algorithm contributed to a more comprehensive analytical context regarding the churn prediction of the proposed model. / Illojalitet prediktering används för att identifiera kunder som är påväg att bli mindre lojala och är ett hjälpsamt verktyg för att ett företag ska kunna driva en konkurrenskraftig verksamhet. I detaljhandel behöves en dynamisk definition av illojalitet för att korrekt kunna identifera illojala kunder. Kundens livstidsvärde är ett mått på monetärt värde av en kundrelation. En avstannad förändring av detta värde indikerar en minskning av kundens lojalitet. Denna rapport föreslår en ny metod för att utföra illojalitet prediktering. Den föreslagna metoden består av ett återkommande neuralt nätverk som används för att identifiera illojalitet hos kunder genom att prediktera kunders livstidsvärde. Resultaten visar att den föreslagna modellen presterar bättre jämfört med slumpmässig metod. Rapporten undersöker också användningen av en k-medelvärdesalgoritm som ett substitut för en regelextraktionsalgoritm. K-medelsalgoritm bidrog till en mer omfattande analys av illojalitet predikteringen.

High Frequency Trading (HFT) algorithms are automated feedback systems interacting with markets to maximize returns on investments. These systems have the potential to read different resolutions of market information at any given time, where Level I information is the minimal information about an equity--essentially its price--and Level II information is the full order book at that time for that equity. This paper presents a study of using Recurrent Neural Network (RNN) models to predict the spread of the DOW Industrial 30 index traded on NASDAQ, using Level I and Level II data as inputs. The results show that Level II data does not significantly improve the prediction of spread when predicting less than 100 millisecond into the future, while it is increasingly informative for spread predictions further into the future. This suggests that HFT algorithms should not attempt to make use of Level II information, and instead reallocate that computation power for improved trading performance, while slower trading algorithms may very well benefit from processing the complete order book.

<p dir="ltr">The research focuses on improving natural language processing (NLP) models by integrating the hierarchical structure of language, which is essential for understanding and generating human language. The main contributions of the study are:</p><ol><li><b>Hierarchical RNN Model:</b> Development of a deep Recurrent Neural Network model that captures both explicit and implicit hierarchical structures in language.</li><li><b>Hierarchical Attention Mechanism:</b> Use of a multi-level attention mechanism to help the model prioritize relevant information at different levels of the hierarchy.</li><li><b>Latent Indicators and Efficient Training:</b> Integration of latent indicators using the Expectation-Maximization algorithm and reduction of computational complexity with Bootstrap sampling and layered training strategies.</li><li><b>Sequence-to-Sequence Model for Translation:</b> Extension of the model to translation tasks, including a novel pre-training technique and a hierarchical decoding strategy to stabilize latent indicators during generation.</li></ol><p dir="ltr">The study claims enhanced performance in various NLP tasks with results comparable to larger models, with the added benefit of increased interpretability.</p>

The automotive industry is inevitably experiencing a paradigm shift from fossil fuels to electric powertrain with significant technological breakthroughs in vehicle electrification. Emerging hybrid electric vehicles were one of the first steps towards cleaner and greener vehicles with a higher fuel economy and lower emission levels. The energy management strategy in hybrid electric vehicles determines the power flow pattern and significantly affects vehicle performance. Therefore, in this thesis, a learning-based strategy is proposed to address the energy management problem of a hybrid electric vehicle in various driving conditions. The idea of a deep recurrent neural network-based energy management strategy is proposed, developed, and evaluated. Initially, a hybrid electric vehicle model with a rule-based supervisory controller is constructed for this case study to obtain training data for the deep recurrent neural network and to evaluate the performance of the proposed energy management strategy. Secondly, due to its capabilities to remember historical data, a long short-term memory recurrent neural network is designed and trained to estimate the powertrain control variables from vehicle parameters. Extensive simulations are conducted to improve the model accuracy and ensure its generalization capability. Also, several hyper-parameters and structures are specifically tuned and debugged for this purpose. The novel proposed energy management strategy takes sequential data as input to capture the characteristics of both driver and controller behaviors and improve the estimation/prediction accuracy. The energy management controller is defined as a time-series problem, and a network predictor module is implemented in the system-level controller of the hybrid electric vehicle model. According to the simulation results, the proposed strategy and prediction model demonstrated lower fuel consumption and higher accuracy compared to other learning-based energy management strategies. / Thesis / Master of Applied Science (MASc)