The transportation industry has a significant effect on the sustainability and development of a society. Learning traffic patterns, and predicting the traffic parameters such as flow or speed for a specific spatiotemporal point is beneficial for transportation systems. For instance, intelligent transportation systems (ITS) can use forecasted results to improve services such as driver assistance systems. Furthermore, the prediction can facilitate urban planning by making management decisions data driven. There are several prediction models for time series regression on traffic data to predict the average speed for different forecasting horizons. In this thesis work, we evaluated Long Short-Term Memory (LSTM), one of the recurrent neural network models and Neural decomposition (ND), a neural network that performs Fourier-like decomposition. The results were compared with the ARIMA model. The persistent model was chosen as a baseline for the evaluation task. We proposed two new criteria in addition to RMSE and r2, to evaluate models for forecasting highly variable velocity changes. The dataset was gathered from highway traffic sensors around the E4 in Stockholm, taken from the “Motorway Control System” (MCS) operated by Trafikverket. Our experiments show that none of the models could predict the highly variable velocity changes at the exact times they happen. The reason was that the adjacent local area had no indications of sudden changes in the average speed of vehicles passing the selected sensor. We also conclude that traditional ML metrics of RMSE and r2 could be augmented with domain specific measures. / Transportbranschen har en betydande inverkan på samhällets hållbarhet och utveckling. Att lära sig trafikmönster och förutsäga trafikparametrar som flöde eller hastighet för en specifik spatio-temporal punkt är fördelaktigt för transportsystem. Intelligenta transportsystem (ITS) kan till exempel använda prognostiserade resultat för att förbättra tjänster som förarassistanssystem. Vidare kan förutsägelsen underlätta stadsplanering genom att göra ledningsbeslut datadrivna. Det finns flera förutsägelsemodeller för tidsserieregression på trafikdata för att förutsäga medelhastigheten för olika prognoshorisonter. I det här avhandlingsarbetet utvärderade vi Långtidsminne (LSTM), en av de återkommande neurala nätverksmodellerna och Neural dekomposition (ND), ett neuralt nätverk som utför Fourierliknande sönderdelning. Resultaten jämfördes med ARIMA-modellen. Den ihållande modellen valdes som utgångspunkt för utvärderingsuppgiften. Vi föreslog två nya kriterier utöver RMSE och r2, för att utvärdera modeller för prognoser av högt variabla hastighetsändringar. Datasetet insamlades från trafiksensor på motorvägar runt E4 i Stockholm, för det så kallade motorvägskontrollsystemet (MCS). Våra experiment visar att ingen av modellerna kan förutsäga de höga variabla hastighetsförändringarna vid exakta tider som de händer. Anledningen var att det intilliggande lokala området inte hade några indikationer på plötsliga förändringar i medelhastigheten hos fordon som passerade den valda sensorn. Vi drar också slutsatsen att traditionella ML-metrics av RMSE och R2 kan kompletteras med domänspecifika åtgärder.

The goal of this project is to compare different machine learning algorithms ability to predict wind power output 48 hours in advance from earlier power data and meteorological wind speed predictions. Three different models were tested, two autoregressive integrated moving average (ARIMA) models one with exogenous regressors one without and one simple LSTM neural net model. It was found that the ARIMA model with exogenous regressors was the most accurate while also beingrelatively easy to interpret and at 1h 45min 32s had a comparatively short training time. The LSTM was less accurate, harder to interpretand took 14h 3min 5s to train. However the LSTM only took 32.7s to create predictions once the model was trained compared to the 33min13.7s it took for the ARIMA model with exogenous regressors to deploy.Because of this fast deployment time the LSTM might be preferable in certain situations. The ARIMA model without exogenous regressors was significantly less accurate than the other two without significantly improving on the other ARIMA model in any way

Les flux de contenus audiovisuels peuvent être représentés sous forme de séquences d’événements (par exemple, des suites d’émissions, de scènes, etc.). Ces données séquentielles se caractérisent par des relations chronologiques pouvant exister entre les événements successifs. Dans le contexte d’une chaîne TV, la programmation des émissions suit une cohérence définie par cette même chaîne, mais peut également être influencée par les programmations des chaînes concurrentes. Dans de telles conditions,les séquences d’événements des flux parallèles pourraient ainsi fournir des connaissances supplémentaires sur les événements d’un flux considéré.La modélisation de séquences est un sujet classique qui a été largement étudié, notamment dans le domaine de l’apprentissage automatique. Les réseaux de neurones récurrents de type Long Short-Term Memory (LSTM) ont notamment fait leur preuve dans de nombreuses applications incluant le traitement de ce type de données. Néanmoins,ces approches sont conçues pour traiter uniquement une seule séquence d’entrée à la fois. Notre contribution dans le cadre de cette thèse consiste à élaborer des approches capables d’intégrer conjointement des données séquentielles provenant de plusieurs flux parallèles.Le contexte applicatif de ce travail de thèse, réalisé en collaboration avec le Laboratoire Informatique d’Avignon et l’entreprise EDD, consiste en une tâche de prédiction du genre d’une émission télévisée. Cette prédiction peut s’appuyer sur les historiques de genres des émissions précédentes de la même chaîne mais également sur les historiques appartenant à des chaînes parallèles. Nous proposons une taxonomie de genres adaptée à de tels traitements automatiques ainsi qu’un corpus de données contenant les historiques parallèles pour 4 chaînes françaises.Deux méthodes originales sont proposées dans ce manuscrit, permettant d’intégrer les séquences des flux parallèles. La première, à savoir, l’architecture des LSTM parallèles(PLSTM) consiste en une extension du modèle LSTM. Les PLSTM traitent simultanément chaque séquence dans une couche récurrente indépendante et somment les sorties de chacune de ces couches pour produire la sortie finale. Pour ce qui est de la seconde proposition, dénommée MSE-SVM, elle permet de tirer profit des avantages des méthodes LSTM et SVM. D’abord, des vecteurs de caractéristiques latentes sont générés indépendamment, pour chaque flux en entrée, en prenant en sortie l’événement à prédire dans le flux principal. Ces nouvelles représentations sont ensuite fusionnées et données en entrée à un algorithme SVM. Les approches PLSTM et MSE-SVM ont prouvé leur efficacité dans l’intégration des séquences parallèles en surpassant respectivement les modèles LSTM et SVM prenant uniquement en compte les séquences du flux principal. Les deux approches proposées parviennent bien à tirer profit des informations contenues dans les longues séquences. En revanche, elles ont des difficultés à traiter des séquences courtes.L’approche MSE-SVM atteint globalement de meilleures performances que celles obtenues par l’approche PLSTM. Cependant, le problème rencontré avec les séquences courtes est plus prononcé pour le cas de l’approche MSE-SVM. Nous proposons enfin d’étendre cette approche en permettant d’intégrer des informations supplémentaires sur les événements des séquences en entrée (par exemple, le jour de la semaine des émissions de l’historique). Cette extension, dénommée AMSE-SVM améliore remarquablement la performance pour les séquences courtes sans les baisser lorsque des séquences longues sont présentées. / In the same way as TV channels, data streams are represented as a sequence of successive events that can exhibit chronological relations (e.g. a series of programs, scenes, etc.). For a targeted channel, broadcast programming follows the rules defined by the channel itself, but can also be affected by the programming of competing ones. In such conditions, event sequences of parallel streams could provide additional knowledge about the events of a particular stream. In the sphere of machine learning, various methods that are suited for processing sequential data have been proposed. Long Short-Term Memory (LSTM) Recurrent Neural Networks have proven its worth in many applications dealing with this type of data. Nevertheless, these approaches are designed to handle only a single input sequence at a time. The main contribution of this thesis is about developing approaches that jointly process sequential data derived from multiple parallel streams. The application task of our work, carried out in collaboration with the computer science laboratory of Avignon (LIA) and the EDD company, seeks to predict the genre of a telecast. This prediction can be based on the histories of previous telecast genres in the same channel but also on those belonging to other parallel channels. We propose a telecast genre taxonomy adapted to such automatic processes as well as a dataset containing the parallel history sequences of 4 French TV channels. Two original methods are proposed in this work in order to take into account parallel stream sequences. The first one, namely the Parallel LSTM (PLSTM) architecture, is an extension of the LSTM model. PLSTM simultaneously processes each sequence in a separate recurrent layer and sums the outputs of each of these layers to produce the final output. The second approach, called MSE-SVM, takes advantage of both LSTM and Support Vector Machines (SVM) methods. Firstly, latent feature vectors are independently generated for each input stream, using the output event of the main one. These new representations are then merged and fed to an SVM algorithm. The PLSTM and MSE-SVM approaches proved their ability to integrate parallel sequences by outperforming, respectively, the LSTM and SVM models that only take into account the sequences of the main stream. The two proposed approaches take profit of the information contained in long sequences. However, they have difficulties to deal with short ones. Though MSE-SVM generally outperforms the PLSTM approach, the problem experienced with short sequences is more pronounced for MSE-SVM. Finally, we propose to extend this approach by feeding additional information related to each event in the input sequences (e.g. the weekday of a telecast). This extension, named AMSE-SVM, has a remarkably better behavior with short sequences without affecting the performance when processing long ones.

Deux aspects importants de la connaissance qu'un individu a pu acquérir par ses expériences correspondent à la mémoire sémantique (celle des connaissances explicites, comme par exemple l'apprentissage de concepts et de catégories décrivant les objets du monde) et la mémoire procédurale (connaissances relatives à l'apprentissage de règles ou de la syntaxe). Cette "mémoire syntaxique" se construit à partir de l'expérience et notamment de l'observation de séquences, suites d'objets dont l'organisation séquentielle obéit à des règles syntaxiques. Elle doit pouvoir être utilisée ultérieurement pour générer des séquences valides, c'est-à-dire respectant ces règles. Cette production de séquences valides peut se faire de façon explicite, c'est-à-dire en évoquant les règles sous-jacentes, ou de façon implicite, quand l'apprentissage a permis de capturer le principe d'organisation des séquences sans recours explicite aux règles. Bien que plus rapide, plus robuste et moins couteux en termes de charge cognitive que le raisonnement explicite, le processus implicite a pour inconvénient de ne pas donner accès aux règles et de ce fait, de devenir moins flexible et moins explicable. Ces mécanismes mnésiques s'appliquent aussi à l'expertise métier : la capitalisation des connaissances pour toute entreprise est un enjeu majeur et concerne aussi bien celles explicites que celles implicites. Au début, l'expert réalise un choix pour suivre explicitement les règles du métier. Mais ensuite, à force de répétition, le choix se fait automatiquement, sans évocation explicite des règles sous-jacentes. Ce changement d'encodage des règles chez un individu en général et particulièrement chez un expert métier peut se révéler problématique lorsqu'il faut expliquer ou transmettre ses connaissances. Si les concepts métiers peuvent être formalisés, il en va en général de tout autre façon pour l'expertise. Dans nos travaux, nous avons souhaité nous pencher sur les séquences de composants électriques et notamment la problématique d’extraction des règles cachées dans ces séquences, aspect important de l’extraction de l’expertise métier à partir des schémas techniques. Nous nous plaçons dans le domaine connexionniste, et nous avons en particulier considéré des modèles neuronaux capables de traiter des séquences. Nous avons implémenté deux réseaux de neurones récurrents : le modèle de Elman et un modèle doté d’unités LSTM (Long Short Term Memory). Nous avons évalué ces deux modèles sur différentes grammaires artificielles (grammaire de Reber et ses variations) au niveau de l’apprentissage, de leurs capacités de généralisation de celui-ci et leur gestion de dépendances séquentielles. Finalement, nous avons aussi montré qu’il était possible d’extraire les règles encodées (issues des séquences) dans le réseau récurrent doté de LSTM, sous la forme d’automate. Le domaine électrique est particulièrement pertinent pour cette problématique car il est plus contraint avec une combinatoire plus réduite que la planification de tâches dans des cas plus généraux comme la navigation par exemple, qui pourrait constituer une perspective de ce travail. / There are two important aspects of the knowledge that an individual acquires through experience. One corresponds to the semantic memory (explicit knowledge, such as the learning of concepts and categories describing the objects of the world) and the other, the procedural or syntactic memory (knowledge relating to the learning of rules or syntax). This "syntactic memory" is built from experience and particularly from the observation of sequences of objects whose organization obeys syntactic rules.It must have the capability to aid recognizing as well as generating valid sequences in the future, i.e., sequences respecting the learnt rules. This production of valid sequences can be done either in an explicit way, that is, by evoking the underlying rules, or implicitly, when the learning phase has made it possible to capture the principle of organization of the sequences without explicit recourse to the rules. Although the latter is faster, more robust and less expensive in terms of cognitive load as compared to explicit reasoning, the implicit process has the disadvantage of not giving access to the rules and thus becoming less flexible and less explicable. These mnemonic mechanisms can also be applied to business expertise. The capitalization of information and knowledge in general, for any company is a major issue and concerns both the explicit and implicit knowledge. At first, the expert makes a choice to explicitly follow the rules of the trade. But then, by dint of repetition, the choice is made automatically, without explicit evocation of the underlying rules. This change in encoding rules in an individual in general and particularly in a business expert can be problematic when it is necessary to explain or transmit his or her knowledge. Indeed, if the business concepts can be formalized, it is usually in any other way for the expertise which is more difficult to extract and transmit.In our work, we endeavor to observe sequences of electrical components and in particular the problem of extracting rules hidden in these sequences, which are an important aspect of the extraction of business expertise from technical drawings. We place ourselves in the connectionist domain, and we have particularly considered neuronal models capable of processing sequences. We implemented two recurrent neural networks: the Elman model and a model with LSTM (Long Short Term Memory) units. We have evaluated these two models on different artificial grammars (Reber's grammar and its variations) in terms of learning, their generalization abilities and their management of sequential dependencies. Finally, we have also shown that it is possible to extract the encoded rules (from the sequences) in the recurrent network with LSTM units, in the form of an automaton. The electrical domain is particularly relevant for this problem. It is more constrained with a limited combinatorics than the planning of tasks in general cases like navigation for example, which could constitute a perspective of this work.

The thesis deals with problems of automatic detection of premature ventricular contractions in ECG records. One detection method which uses a convolutional neural network and LSTM units is implemented in the Python language. Cardiac cycles extracted from one-lead ECG were used for detection. F1 score for binary classification (PVC and normal beat) on the test dataset reached 96,41 % and 81,76 % for three-class classification (PVC, normal beat and other arrhythmias). Lastly, the accuracy of the classification is evaluated and discussed, the achieved results for binary classification are comparable to the results of methods described in different papers.

This thesis deals with anomaly detection of log data. Big software systems produce a great amount of log data which are not further processed. There are usually so many logs that it becomes impossible to check every log entry manually. In this thesis we introduce models that minimize primarily count of false positive predictions with expected complexity of data annotation taken into account. The compared models are based on PCA algorithm, N-gram model and recurrent neural networks with LSTM cell. In the thesis we present results of the models on widely used datasets and also on a real dataset provided by HAVIT, s.r.o. 1

System log files are filled with logged events, status codes, and other messages. By analyzing the log files, the systems current state can be determined, and find out if something during its execution went wrong. Log file analysis has been studied for some time now, where recent studies have shown state-of-the-art performance using machine learning techniques. In this thesis, document classification solutions were tested on log files in order to classify regular system runs versus abnormal system runs. To solve this task, supervised and unsupervised learning methods were combined. Doc2Vec was used to extract document features, and Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) based architectures on the classification task. With the use of the machine learning models and preprocessing techniques the tested models yielded an f1-score and accuracy above 95% when classifying log files.

The success of Internet of Things solutions allowed the establishment of new applications such as smart hydroponic agriculture. One typical problem in such an application is the rapid degradation of the deployed sensors. Traditionally, this problem is resolved by frequent manual maintenance, which is considered to be ineffective and may harm the crops in the long run. The main purpose of this thesis was to propose a machine learning approach for automating the detection of sensor fault drifts. In addition, the solution’s operability was investigated in a cloud computing environment in terms of the response time. This thesis proposes a detection algorithm that utilizes RNN in predicting sensor drifts from time-series data streams. The detection algorithm was later named; Predictive Sliding Detection Window (PSDW) and consisted of both forecasting and classification models. Three different RNN algorithms, i.e., LSTM, CNN-LSTM, and GRU, were designed to predict sensor drifts using forecasting and classification techniques. The algorithms were compared against each other in terms of relevant accuracy metrics for forecasting and classification. The operability of the solution was investigated by developing a web server that hosted the PSDW algorithm on an AWS computing instance. The resulting forecasting and classification algorithms were able to make reasonably accurate predictions for this particular scenario. More specifically, the forecasting algorithms acquired relatively low RMSE values as ~0.6, while the classification algorithms obtained an average F1-score and accuracy of ~80% but with a high standard deviation. However, the response time was ~5700% slower during the simulation of the HTTP requests. The obtained results suggest the need for future investigations to improve the accuracy of the models and experiment with other computing paradigms for more reliable deployments.

In this thesis, we analyse the sentiment in equity research reports written by analysts at Skandinaviska Enskilda Banken (SEB). We provide a description of established statistical and machine learning methods for classifying the sentiment in text documents as positive or negative. Specifically, a form of recurrent neural network known as long short-term memory (LSTM) is of interest. We investigate two different labelling regimes for generating training data from the reports. Benchmark classification accuracies are obtained using logistic regression models. Finally, two different word embedding models and bidirectional LSTMs of varying network size are implemented and compared to the benchmark results. We find that the logistic regression works well for one of the labelling approaches, and that the best LSTM models outperform it slightly. / I denna rapport analyserar vi sentimentet, eller attityden, i aktieanalysrapporter skrivna av analytiker på Skandinaviska Enskilda Banken (SEB). Etablerade statistiska metoder och maskininlärningsmetoder för klassificering av sentimentet i textdokument som antingen positivt eller negativt presenteras. Vi är speciellt intresserade av en typ av rekurrent neuronnät känt som long short-term memory (LSTM). Vidare undersöker vi två olika scheman för att märka upp träningsdatan som genereras från rapporterna. Riktmärken för klassificeringsgraden erhålls med hjälp av logistisk regression. Slutligen implementeras två olika ordrepresentationsmodeller och dubbelriktad LSTM av varierande nätverksstorlek, och jämförs med riktmärkena. Vi finner att logistisk regression presterar bra för ett av märkningsschemana, och att LSTM har något bättre prestanda.

[pt] Prever precisamente a produção de óleo é essencial para o planejamento e administração de um reservatório. Entretanto, prever a produção de óleo é um problema complexo e não linear, devido a todas as propriedades geofísicas que com pequenas variações podem resultar em differentes cenários. Além disso, todas as decisões tomadas durante a exploração do projeto devem considerar diferentes algoritmos para simular dados, fornecer cenários e conduzir a boas deduções. Para reduzir as incertezas nas simulações, estudos recentes propuseram o uso de algoritmos de aprendizado de maquina para solução de problemas da engenharia de reservatórios, devido a capacidade desses modelos de extrair o maxiomo de informações de um conjunto de dados. Essa tese propôe o uso ed duas tecnicas de machine learning para prever a produção diaria de óleo de um reservatório. Inicialmente, a produção diária de óleo é considerada uma série temporal, é pré-processada e reestruturada como um problema de aprendizado supervisionado. O modelo Random Forest, uma extensão das arvores de decisão muito utilizado em problemas de regressão e classificação, é utilizado para predizer um passo de tempo a frente. Entretanto, as restrições dessa abordagem nos conduziram a um modelo mais robusto, as redes neurais recorrentes LSTM, que são utilizadas em varios estudos como uma ferramenta dee aprendizado profundo adequada para modelagem de séries temporais. Várias configurações de redes LSTM foram construidas para implementar a previsão de um passo de tempo e de multiplos passos de tempo, a pressão do fundo de poço foi incorporada aos dados de entrada. Para testar a eficacia dos modelos propostos, foram usados quatro conjunto de dados diferentes, três gerados sintéticamente e um conjunto de dados reais do campo de produção VOlve, como casos de estudo para conduzir os experimentos. Os resultados indicam que o Random Forest é suficiente para previsões de um passo de tempo da produção de óleo e o LSTM é capaz de lidar com mais dados de entrada e estimar multiplos passos de tempo da produção de óleo. / [en] Precisely forecasting oil field performance is essential in oil reservoir planning and management. Nevertheless, forecasting oil production is a complex nonlinear problem due to all geophysical and petrophysical properties that may result in different effects with a bit of change. Thus, all decisions to be made during an exploitation project must consider different efficient algorithms to simulate data, providing robust scenarios to lead to the best deductions. To reduce the uncertainty in the simulation process, recent studies have efficiently introduced machine learning algorithms for solving reservoir engineering problems since they can extract the maximum information from the dataset. This thesis proposes using two machine learning techniques to predict the daily oil production of an offshore reservoir. Initially, the oil rate production is considered a time series and is pre-processed and restructured to fit a supervised learning problem. The Random Forest model is used to forecast a one-time step, which is an extension of decision tree learning, widely used in regression and classification problems for supervised machine learning. Regardless, the restrictions of this approach lead us to a more robust model, the LSTM RNN s, which are proposed by several studies as a suitable deep learning technique for time series modeling. Various configurations of LSTM RNN s were constructed to implement single-step and multi-step oil rate forecasting and down-hole pressure was incorporated to the inputs. For testing the robustness of the proposed models, we use four different datasets, three of them synthetically generated and one from a public real dataset, the Volve oil field, as a case study to conduct the experiments. The results indicate that the Random Forest model could sufficiently estimate the one-time step of the oil field production, and LSTM could handle more inputs and adequately estimate multiple-time steps of oil production.