Modern automobiles have been proven vulnerable to hacking by security researchers. By exploiting vulnerabilities in the car's external interfaces, attackers can access a car's controller area network (CAN) bus and cause malicious effects. We seek to detect these attacks on the bus as a last line of defence against automotive cyber attacks. The CAN bus standard defines a low-level message structure, upon which manufacturers layer their own proprietary command protocols; attacks must similarly be tailored for their target. This variability makes intrusion detection methods difficult to apply to the automotive CAN bus. Nevertheless, the bus traffic is generated by machines; thus we hypothesize that it can be characterized with machine learning, and that attacks produce anomalous traffic. Our goals are to show that anomaly detection trained without understanding of the message contents can detect attacks, and to create a framework for understanding how the characteristics of a novel attack can be used to predict its detectability. We developed a model that describes attacks based on their effect on bus traffic, informed by a review of published material on car hacking in combination with analysis of CAN traffic from a 2012 Subaru Impreza. The model specifies three high-level categories of effects: attacks that insert foreign packets, attacks that affect packet timing, and attacks that only modify data within packets. Foreign packet attacks are trivially detectable. For timing-based anomalies, we developed features suitable for one-class classification methods. For packet stream data word anomalies, we adapted recurrent neural networks and multivariate Markov model methods to sequence anomaly detection and compared their performance. We conducted experiments to evaluate our detection methods with special attention to the trade-off between precision and recall, given that a practical system requires a very low false alarm rate. The methods were evaluated by synthesizing anomalies within each attack category, parameterized to adjust their covertness. We generalize from the results to enable prediction of detection rates for new attacks using these methods.

This study is about prediction of the stockmarket through a comparison of neural networks and statistical models. The study aims to improve the accuracy of stock prediction. Much of the research made on predicting shares deals with statistical models, but also neural networks and then mainly the types RNN and CNN. No research has been done on how these neural networks can be combined, which is why this study aims for this. Tests are made on statistical models, neural networks and combined neural networks to predict stocks at minute level. The result shows that a combination of two neural networks of type RNN gives the best accuracy in the prediction of shares. The accuracy of the predictions increases further if these combined neural networks are trained to predict different time horizons. In addition to tests for accuracy, simulations have also been made which also confirm that there is some possibility to predict shares. Two combined RNNs gave best results, but in the simulations, even CNN made good predictions. One conclusion can be drawn that the stock market is not entirely effective as some opportunity to predict future values exists. Another conclusion is that neural networks are better than statistical models to predict stocks if the neural networks are combined and are of type RNN. / Denna studie behandlar prediktion av aktier genom en jämförelse av neurala nätverk och statistiska modeller. Studien syftar till att förbättra noggrannheten för aktieprediktion. Mycket av den forskning som gjorts om att förutspå aktier behandlar statistiska modeller, men även neurala nätverk och då främst typerna RNN och CNN. Ingen forskning har dock gjorts på hur dessa neurala nätverk kan kombineras, varför denna studie syftar till just detta. Tester är gjorda på statistiska modeller, neurala nätverk och kombinerade neurala nätverk för att förutspå aktier på minutnivå. Resultatet visar att en kombination av två neurala nätverk av typen RNN ger bäst noggrannhet vid prediktion av aktier. Noggrannheten i prediktionerna ökar ytterligare om dessa neurala nätverk tränas för att förutspå olika tidshorisont. Utöver tester för prediktionernas noggrannhet har även simuleringar genomförts som även de bekräftar att viss möjlighet finns att förutspå aktier. Två kombinerade RNN gav bra resultat, men här visade även CNN bra prediktioner. En slutsats kan dras om att aktiemarknaden inte är helt effektiv då viss möjlighet att förutspå framtida värden finns. Ytterligare en slutsats är att neurala nätverk är bättre än statistiska modeller till att förutspå aktier om de neurala nätverken kombineras och är av typen RNN.

Background. This thesis focuses on the task of historical document semantic segmentation with recurrent neural networks. Document semantic segmentation involves the segmentation of a page into different meaningful regions and is an important prerequisite step of automated document analysis and digitisation with optical character recognition. At the time of writing, convolutional neural network based solutions are the state-of-the-art for analyzing document images while the use of recurrent neural networks in document semantic segmentation has not yet been studied. Considering the nature of a recurrent neural network and the recent success of recurrent neural networks in document image binarization, it should be possible to employ a recurrent neural network for document semantic segmentation and further achieve high performance results. Objectives. The main objective of this thesis is to investigate if recurrent neural networks are a viable alternative to convolutional neural networks in document semantic segmentation. By using a combination of a convolutional neural network and a recurrent neural network, another objective is also to determine if the performance of the combination can improve upon the existing case of only using the recurrent neural network. Methods. To investigate the impact of recurrent neural networks in document semantic segmentation, three different recurrent neural network architectures are implemented and trained while their performance are further evaluated with Intersection over Union. Afterwards their segmentation result are compared to a convolutional neural network. By performing pre-processing on training images and multi-class labeling, prediction images are ultimately produced by the employed models. Results. The results from the gathered performance data shows a 2.7% performance difference between the best recurrent neural network model and the convolutional neural network. Notably, it can be observed that this recurrent neural network model has a more consistent performance than the convolutional neural network but comparable performance results overall. For the other recurrent neural network architectures lower performance results are observed which is connected to the complexity of these models. Furthermore, by analyzing the performance results of a model using a combination of a convolutional neural network and a recurrent neural network, it can be noticed that the combination performs significantly better with a 4.9% performance increase compared to the case with only using the recurrent neural network. Conclusions. This thesis concludes that recurrent neural networks are likely a viable alternative to convolutional neural networks in document semantic segmentation but that further investigation is required. Furthermore, by combining a convolutional neural network with a recurrent neural network it is concluded that the performance of a recurrent neural network model is significantly increased. / Bakgrund. Detta arbete handlar om semantisk segmentering av historiska dokument med recurrent neural network. Semantisk segmentering av dokument inbegriper att dela in ett dokument i olika regioner, något som är viktigt för att i efterhand kunna utföra automatisk dokument analys och digitalisering med optisk teckenläsning. Vidare är convolutional neural network det främsta alternativet för bearbetning av dokument bilder medan recurrent neural network aldrig har använts för semantisk segmentering av dokument. Detta är intressant eftersom om vi tar hänsyn till hur ett recurrent neural network fungerar och att recurrent neural network har uppnått mycket bra resultat inom binär bearbetning av dokument, borde det likväl vara möjligt att använda ett recurrent neural network för semantisk segmentering av dokument och även här uppnå bra resultat. Syfte. Syftet med arbetet är att undersöka om ett recurrent neural network kan uppnå ett likvärdigt resultat jämfört med ett convolutional neural network för semantisk segmentering av dokument. Vidare är syftet även att undersöka om en kombination av ett convolutional neural network och ett recurrent neural network kan ge ett bättre resultat än att bara endast använda ett recurrent neural network. Metod. För att kunna avgöra om ett recurrent neural network är ett lämpligt alternativ för semantisk segmentering av dokument utvärderas prestanda resultatet för tre olika modeller av recurrent neural network. Därefter jämförs dessa resultat med prestanda resultatet för ett convolutional neural network. Vidare utförs förbehandling av bilder och multi klassificering för att modellerna i slutändan ska kunna producera mätbara resultat av uppskattnings bilder. Resultat. Genom att utvärdera prestanda resultaten för modellerna kan vi i en jämförelse med den bästa modellen och ett convolutional neural network uppmäta en prestanda skillnad på 2.7%. Noterbart i det här fallet är att den bästa modellen uppvisar en jämnare fördelning av prestanda. För de två modellerna som uppvisade en lägre prestanda kan slutsatsen dras att deras utfall beror på en lägre modell komplexitet. Vidare vid en jämförelse av dessa två modeller, där den ena har en kombination av ett convolutional neural network och ett recurrent neural network medan den andra endast har ett recurrent neural network uppmäts en prestanda skillnad på 4.9%. Slutsatser. Resultatet antyder att ett recurrent neural network förmodligen är ett lämpligt alternativ till ett convolutional neural network för semantisk segmentering av dokument. Vidare dras slutsatsen att en kombination av de båda varianterna bidrar till ett bättre prestanda resultat.

Tracab’s optical tracking system allows to track the 2-dimensional trajectories of players and ball during a football game. Using this data it is possible to train machine learning models to identify events that happen during the match. In this thesis, we explore the detection of corners, free kicks, and throw-in events by means of neural networks. Training a model to solve this task is not easy; the neural network needs to model the spatio-temporal interactions between different agents moving in a 2-dimensional space. We decided to address this problem using graph neural networks in combination with recurrent neural networks, which allow us to model respectively the spatial and temporal components of the data. Tracking the position of the ball is difficult, which makes the dataset noisy. In this thesis, we mainly work with a version of the dataset where the position of the ball has been manually corrected. However, to study how the noisy position of the ball affects the results we also train the models on the original data. The results show that detecting the corner and the throw-in is much easier than detecting the free kick. Moreover, the noisy position of the ball affects significantly the performance of the model. We conclude that to train the model on the original data it is necessary to use a much larger training set. Since the amount of training data for these events is limited, we also train the model on the more generic ball-dead-to-alive event, for which much more data is available, and we observe that by increasing the amount of training data the results can improve significantly. In this report, we also provide an in-depth discussion about all the challenges faced during the project and how different hyperparameters and design choices can affect the results. / Tracabs optiska spårningssystem gör det möjligt att spåra de 2-dimensionella banorna för spelare och boll under en fotbollsmatch. Med hjälp av dessa data är det möjligt att träna maskininlärningsmodeller för att identifiera händelser som inträffar under matchen. I denna avhandling utforskar vi upptäckten av hörnor, frisparkar och inkastningshändelser med hjälp av neurala nätverk. Att träna en modell för att lösa denna uppgift är inte lätt; det neurala nätverket behöver modellera de rums-temporala interaktionerna mellan olika agenter som rör sig i ett 2-dimensionellt rum. Vi bestämde oss för att ta itu med detta problem med hjälp av grafiska neurala nätverk i kombination med återkommande neurala nätverk, vilket gör att vi kan modellera de rumsliga respektive temporala komponenterna i datan. Det är svårt att spåra bollens position, vilket gör datauppsättningen bullrig. I detta examensarbete arbetar vi främst med en version av datamängden där bollens position har korrigerats manuellt. Men för att studera hur bollens bullriga position påverkar resultaten tränar vi också modellerna på originaldata. Resultaten visar att det är mycket lättare att upptäcka hörna och inkastet än att upptäcka frisparken. Dessutom påverkar bollens bullriga position avsevärt modellens prestanda. Vi drar slutsatsen att för att träna modellen på originaldata är det nödvändigt att använda en mycket större träningsuppsättning. Eftersom mängden träningsdata för dessa evenemang är begränsad, tränar vi också modellen på den mer generiska bollen död-till-levande-händelsen, för vilken mycket mer data finns tillgänglig, och vi observerar att genom att öka mängden träningsdata resultaten kan förbättras avsevärt. I denna rapport ger vi också en fördjupad diskussion om alla utmaningar som ställs inför under projektet och hur olika hyperparametrar och designval kan påverka resultaten.

Major advances in machine learning have made image recognition applications, with Artificial Neural Network, blossom over the recent years. The aim of this thesis was to find a solution to recognize digits from a number sequence on an ID tag, used to identify farm animals, with the help of image recognition. A Recurrent Convolutional Neural Network solution called PPNet was proposed and tested on a data set called Animal Identification Tags. A transfer learning method was also used to test if it could help PPNet generalize and better recognize digits. PPNet was then compared against Microsoft Azures own image recognition API, to determine how PPNet compares to a general solution. PPNet, while not performing as good, still managed to achieve competitive results to the Azure API.

Knowledge Graph is a rising topic in the field of Artificial Intelligence. As the current trend of knowledge representation, Knowledge graph research is utilizing the large knowledge base freely available on the internet. Knowledge graph also allows inspection, analysis, the reasoning of all knowledge in reality. To enable the ambitious idea of modeling the knowledge of the world, different theory and implementation emerges. Nowadays, we have the opportunity to use freely available information from Wikipedia and Wikidata. The thesis investigates and formulates a theory about learning from Knowledge Graph. The thesis researches probabilistic knowledge graph. It only focuses on a branch called latent feature models in learning probabilistic knowledge graph. These models aim to predict possible relationships of connected entities and relations. There are many models for such a task. The metrics and training process is detailed described and improved in the thesis work. The efficiency and correctness enable us to build a more complex model with confidence. The thesis also covers possible problems in finding and proposes future work.

Although nonlinear model predictive control (NMPC) might be the best choice for a nonlinear plant, it is still not widely used. This is mainly due to the computational burden associated with solving online a set of nonlinear differential equations and a nonlinear dynamic optimization problem in real time. This thesis is concerned with strategies aimed at reducing the computational burden involved in different stages of the NMPC such as optimization problem, state estimation, and nonlinear model identification. A major part of the computational burden comes from function and derivative evaluations required in different parts of the NMPC algorithm. In this work, the problem is tackled using a recently introduced efficient tool, the automatic differentiation (AD). Using the AD tool, a function is evaluated together with all its partial derivative from the code defining the function with machine accuracy. A new NMPC algorithm based on nonlinear least square optimization is proposed. In a first–order method, the sensitivity equations are integrated using a linear formula while the AD tool is applied to get their values accurately. For higher order approximations, more terms of the Taylor expansion are used in the integration for which the AD is effectively used. As a result, the gradient of the cost function against control moves is accurately obtained so that the online nonlinear optimization can be efficiently solved. In many real control cases, the states are not measured and have to be estimated for each instance when a solution of the model equations is needed. A nonlinear extended version of the Kalman filter (EKF) is added to the NMPC algorithm for this purpose. The AD tool is used to calculate the required derivatives in the local linearization step of the filter automatically and accurately. Offset is another problem faced in NMPC. A new nonlinear integration is devised for this case to eliminate the offset from the output response. In this method, an integrated disturbance model is added to the process model input or output to correct the plant/model mismatch. The time response of the controller is also improved as a by–product. The proposed NMPC algorithm has been applied to an evaporation process and a two continuous stirred tank reactor (two–CSTR) process with satisfactory results to cope with large setpoint changes, unmeasured severe disturbances, and process/model mismatches. When the process equations are not known (black–box) or when these are too complicated to be used in the controller, modelling is needed to create an internal model for the controller. In this thesis, a continuous time recurrent neural network (CTRNN) in a state–space form is developed to be used in NMPC context. An efficient training algorithm for the proposed network is developed using AD tool. By automatically generating Taylor coefficients, the algorithm not only solves the differentiation equations of the network but also produces the sensitivity for the training problem. The same approach is also used to solve online the optimization problem of the NMPC. The proposed CTRNN and the predictive controller were tested on an evaporator and two–CSTR case studies. A comparison with other approaches shows that the new algorithm can considerably reduce network training time and improve solution accuracy. For a third case study, the ALSTOM gasifier, a NMPC via linearization algorithm is implemented to control the system. In this work a nonlinear state–space class Wiener model is used to identify the black–box model of the gasifier. A linear model of the plant at zero–load is adopted as a base model for prediction. Then, a feedforward neural network is created as the static gain for a particular output channel, fuel gas pressure, to compensate its strong nonlinear behavior observed in open–loop simulations. By linearizing the neural network at each sampling time, the static nonlinear gain provides certain adaptation to the linear base model. The AD tool is used here to linearize the neural network efficiently. Noticeable performance improvement is observed when compared with pure linear MPC. The controller was able to pass all tests specified in the benchmark problem at all load conditions.

No description available.

The context of this report and the IRIDIA laboratory are described in the preface. Evolutionary Robotics and the box-pushing task are presented in the introduction.The building of a test system supporting Evolutionary Robotics experiments is then detailed. This system is made of a robot simulator and a Genetic Algorithm. It is used to explore the possibility of evolving box-pushing behaviours. The bootstrapping problem is explained, and a novel approach for dealing with it is proposed, with results presented.Finally, ideas for extending this approach are presented in the conclusion.

Text classification has always been a concern in area of natural language processing, especially nowadays the data are getting massive due to the development of internet. Recurrent neural network (RNN) is one of the most popular method for natural language processing due to its recurrent architecture which give it ability to process serialized information. In the meanwhile, Convolutional neural network (CNN) has shown its ability to extract features from visual imagery. This paper combine the advantages of RNN and CNN and proposed a model called BLSTM-C for Chinese text classification. BLSTM-C begins with a Bidirectional long short-term memory (BLSTM) layer which is an special kind of RNN to get a sequence output based on the past context and the future context. Then it feed this sequence to CNN layer which is utilized to extract features from the previous sequence. We evaluate BLSTM-C model on several tasks such as sentiment classification and category classification and the result shows our model’s remarkable performance on these text tasks.