<div>The air traffic system is one of the most complex and safety-critical systems, which is expected to grow at an average rate of 0.9% a year -- from 51.8 million operational activities in 2018 to 62 million in 2039 -- within the National Airspace System. In such systems, it is important to identify degradations in system performance, especially in terms of safety and efficiency. Among the operations of various subsystems of the air traffic system, the arrival and departure operations in the terminal airspace require more attention because of its higher impact (about 75% incidents) on the entire system's safety, ranging from single aircraft incidents to multi-airport congestion incidents.</div><div><br></div><div>The first goal of this dissertation is to identify the air traffic system's degradations -- called anomalies -- in the multi-airport terminal airspace or metroplex airspace, by developing anomaly detection models that can separate anomalous flights from normal ones. Within the metroplex airspace, airport operational parameters such as runway configuration and coordination between proximal airports are a major driving factor in aircraft’s behaviors. As a substantial amount of data is continually recording such behaviors through sensing technologies and data collection capabilities, modern machine learning techniques provide powerful tools for the identification of anomalous flights in the metroplex airspace. The proposed algorithm ingests heterogeneous data, comprising the surveillance dataset, which represents an aircraft’s physical behaviors, and the airport operations dataset, which reflects operational procedures at airports. Typically, such aviation data is unlabeled, and thus the proposed algorithm is developed based on hierarchical unsupervised learning approaches for anomaly detection. This base algorithm has been extended to an anomaly monitoring algorithm that uses the developed anomaly detection models to detect anomalous flights within real-time streaming data.</div><div><br></div><div>A natural next-step after detecting anomalies is to determine the causes for these anomalies. This involves identifying the occurrence of precursors, which are triggers or conditions that precede an anomaly and have some operational correlation to the occurrence of the anomaly. A precursor detection algorithm is developed which learns the causes for the detected anomalies using supervised learning approaches. If detected, the precursor could be used to trigger actions to avoid the anomaly from ever occurring.</div><div><br></div><div>All proposed algorithms are demonstrated with real air traffic surveillance and operations datasets, comprising of departure and arrival operations at LaGuardia Airport, John F. Kennedy International Airport, and Newark Liberty International Airport, thereby detecting and predicting anomalies for all airborne operations in the terminal airspace within the New York metroplex. Critical insight regarding air traffic management is gained from visualizations and analysis of the results of these extensive tests, which show that the proposed algorithms have a potential to be used as decision-support tools that can aid pilots and air traffic controllers to mitigate anomalies from ever occurring, thus improving the safety and efficiency of metroplex airspace operations.</div>

Orientador: Kelton Augusto Pontara da Costa / Resumo: À medida que a Internet cresce com o passar dos anos, é possível observar um aumento na quantidade de dados que trafegam nas redes de computadores do mundo todo. Em um contexto onde o volume de dados encontra-se em constante renovação, sob a perspectiva da área de Segurança de Redes de Computadores torna-se um grande desafio assegurar, em termos de eficácia e eficiência, os sistemas computacionais da atualidade. Dentre os principais mecanismos de segurança empregados nestes ambientes, destacam-se os Sistemas de Detecção de Intrusão em Rede. Muito embora a abordagem de detecção por assinatura seja suficiente no combate de ataques conhecidos nessas ferramentas, com a eventual descoberta de novas vulnerabilidades, faz-se necessário a utilização de abordagens de detecção por anomalia para amenizar o dano de ataques desconhecidos. No campo acadêmico, diversos trabalhos têm explorado o desenvolvimento de abordagens híbridas com o intuito de melhorar a acurácia dessas ferramentas, com o auxílio de técnicas de Aprendizado de Máquina. Nesta mesma linha de pesquisa, o presente trabalho propõe a aplicação destas técnicas para a detecção de intrusão em um ambiente tempo real mediante uma ferramenta popular e amplamente utilizada, o Snort. Os resultados obtidos mostram que em determinados cenários de ataque, a abordagem de detecção baseada em anomalia pode se sobressair em relação à abordagem de detecção baseada em assinatura, com destaque às técnicas AdaBoost, Florestas Aleatórias, Árvor... (Resumo completo, clicar acesso eletrônico abaixo) / Abstract: As the Internet grows over the years, it is possible to observe an increase in the amount of data that travels on computer networks around the world. In a context where data volume is constantly being renewed, from the perspective of the Network Security area it becomes a great challenge to ensure, in terms of effectiveness and efficiency, today’s computer systems. Among the main security mechanisms employed in these environments, stand out the Network Intrusion Detection Systems. Although the signature-based detection approach is sufficient to combat known attacks in these tools, with the eventual discovery of new vulnerabilities, it is necessary to use anomaly-based detection approaches to mitigate the damage of unknown attacks. In the academic field, several works have explored the development of hybrid approaches in order to improve the accuracy of these tools, with the aid of Machine Learning techniques. In this same line of research, the present work proposes the application of these techniques for intrusion detection in a real time environment using a popular and widely used tool, the Snort. The obtained results shows that in certain attack scenarios, the anomaly-based detection approach may outperform the signature-based detection approach, with emphasis on the techniques AdaBoost, Random Forests, Decision Tree and Linear Support Vector Machine. / Mestre

With the explosion of the number of distributed applications, a new dynamic server environment emerged grouping servers into clusters, whose utilization depends on the cur- rent demand for the application. Detecting and fixing erratic server behavior is paramount for providing maximal service stability and availability. Using standard techniques to de- tect such behavior is yielding sub-optimal results. We have collected a dataset of OS-level performance metrics from a cluster running a streaming distributed application and in- jected artificially created anomalies. We then selected a set of various machine learning algorithms and trained them for anomaly detection on said dataset. We evaluated the algorithms performance and proposed a system for generating notifications of possible erratic behavior, based on the analysis of the best performing algorithm. 1

Context: Maritime Surveillance (MS) has received increased attention from a civilian perspective in recent years. Anomaly detection (AD) is one of the many techniques available for improving the safety and security in the MS domain. Maritime authorities utilize various confidential data sources for monitoring the maritime activities; however, a paradigm shift on the Internet has created new sources of data for MS. These newly identified data sources, which provide publicly accessible data, are the open data sources. Taking advantage of the open data sources in addition to the traditional sources of data in the AD process will increase the accuracy of the MS systems. Objectives: The goal is to investigate the potential open data as a complementary resource for AD in the MS domain. To achieve this goal, the first step is to identify the applicable open data sources for AD. Then, a framework for AD based on the integration of open and closed data sources is proposed. Finally, according to the proposed framework, an AD system with the ability of using open data sources is developed and the accuracy of the system and the validity of its results are evaluated. Methods: In order to measure the system accuracy, an experiment is performed by means of a two stage random sampling on the vessel traffic data and the number of true/false positive and negative alarms in the system is verified. To evaluate the validity of the system results, the system is used for a period of time by the subject matter experts from the Swedish Coastguard. The experts check the detected anomalies against the available data at the Coastguard in order to obtain the number of true and false alarms. Results: The experimental outcomes indicate that the accuracy of the system is 99%. In addition, the Coastguard validation results show that among the evaluated anomalies, 64.47% are true alarms, 26.32% are false and 9.21% belong to the vessels that remain unchecked due to the lack of corresponding data in the Coastguard data sources. Conclusions: This thesis concludes that using open data as a complementary resource for detecting anomalous behavior in the MS domain is not only feasible but also will improve the efficiency of the surveillance systems by increasing the accuracy and covering some unseen aspects of maritime activities. / This thesis investigated the potential open data as a complementary resource for Anomaly Detection (AD) in the Maritime Surveillance (MS) domain. A framework for AD was proposed based on the usage of open data sources along with other traditional sources of data. According to the proposed AD framework and the algorithms for implementing the expert rules, the Open Data Anomaly Detection System (ODADS) was developed. To evaluate the accuracy of the system, an experiment on the vessel traffic data was conducted and an accuracy of 99% was obtained for the system. There was a false negative case in the system results that decreased the accuracy. It was due to incorrect AIS data in a special situation that was not possible to be handled by the detection rules in the scope of this thesis. The validity of the results was investigated by the subject matter experts from the Swedish Coastguard. The validation results showed that the majority of the ODADS evaluated anomalies were true alarms. Moreover, a potential information gap in the closed data sources was observed during the validation process. Despite the high number of true alarms, the number of false alarms was also considerable that was mainly because of the inaccurate open data. This thesis provided insights into the open data as a complement to the common data sources in the MS domain and is concluded that using open data will improve the efficiency of the surveillance systems by increasing the accuracy and covering some unseen aspects of maritime activities.

Manually analysing logfiles is a very time consuming and error-prone effort. By developing a system to automatically analysing the logfiles it is possible to both increase the speed and accuracy of the analysis. This thesis presents a method for automatic anomaly detection in logfiles using statistical analysis and threshold based classification. The presented method uses five different threshold based approaches to identify anomalous entries within a logfile. Each of the five approaches was successful in identifying and reporting perceived anomalies within 805 logfiles provided by Sandvine, it was however not possible to do a formal evaluation of the results due to a lack of a ground truth.

Gripen E, a fighter jet developed by Saab, has to fulfill a number of specifications and is therefore tested thoroughly. This project is about detecting anomalies in such tests and thereby improving the automation of the test data evaluation. The methodology during this project was to model the expected deviation between the measured signals and the corresponding signals from a fuel system model using machine learning methods. This methodology was applied to the mass in one of the fuel tanks. The challenge lies in the fact that the expected deviation is unknown and dependent on the operating conditions of the fuel system in the aircraft. Furthermore, two different machine learning approaches to estimate a prediction interval, within which the residual was expected to be, were tested. These were quantile regression and a variance estimation based method. The machine learning models used in this project were LSTM, Ridge Regression, Random Forest Regressor and Gradient Boosting Regressor. One of the problems encountered was imbalanced data, since different operating modes were not equally represented. Also, whether the time dependency of the signals had to be taken into account was investigated. Moreover, choosing which input signals to use for the machine learning methods had a large impact on the result. The concept appears to work well. Known anomalies were detected, and with a low degree of false alarms. The variance estimation based approach seems superior to quantile regression. For data containing anomalies, the target signal drifted away significantly outside the boundaries of the prediction interval. Such test flights were flagged for anomaly. Furthermore, the concept was also successfully verified for another fuel tank, with only minor and obvious adaptations, such as replacing the target signal with the new one.

The purpose of this thesis is to investigate how data from a residential property owner can be utilized to enable better energy management for their building stock. Specifically, this is done through the development of two machine learning models with the objective of detecting anomalies in the existing data of electricity consumption. The dataset consists of two years of residential electricity consumption for 193 substations belonging to the residential property owner Uppsalahem. The first of the developed models uses the K-means method to cluster substations with similar consumption patterns to create electricity profiles, while the second model uses Gaussian process regression to predict electricity consumption of a 24 hour timeframe. The performance of these models is evaluated and the optimal models resulting from this process are implemented to detect anomalies in the electricity consumption data. Two different algorithms for anomaly detection are presented, based on the differing properties of the two earlier models. During the evaluation of the models, it is established that the consumption patterns of the substations display a high variability, making it difficult to accurately model the full dataset. Both models are shown to be able to detect anomalies in the electricity consumption data, but the K-means based anomaly detection model is preferred due to it being faster and more reliable. It is concluded that substation electricity consumption is not ideal for anomaly detection, and that if a model should be implemented, it should likely exclude some of the substations with less regular consumption profiles.

No description available.

Insider attacks aiming at stealing data are highly common, according to recent studies, and they are carried out in precise patterns. In order to protect against these threats, additional security measures, such as access control and encryption, must be used in conjunction with tools and methods that can detect anomalies in data access. By analyzing the input query syntax and the amount of data returned in the responses, we can deduce individuals' access patterns. Our method is based on SQL queries in database log files, which allow us to build profiles of ordinary users' access behavior by their doctors. Anomalies that deviate from these characteristics are deemed anomalous and thus indicative of possible data exfiltration or misuse. This paper uses machine learning techniques in existing algorithms to detect outliers and aggregate related data into clusters. Due to the sensitivity of the real-world data and restricting access to such datasets, we have developed our logfiles that groups log lines sequentially based on time and access intervals. Generated log files containing known abnormalities are used to demonstrate the use of real datasets. Our findings demonstrate that our method can effectively detect these anomalies, albeit more research by specialists is required to ensure whether the abnormalities detected were appropriately recognized.

The global micromobility market is a fast growing market valued at USD 40.19 Billion in 2020. As the market grows, it is of great importance for companies to gain market shares in order to stay competitive and be the first choice within micromobility services. This can be achieved by, e.g., offering a safe micromobility service, for both riders and other road users. With state-of-the-art technology, accident prevention and preventing misuse of scooters and cities’ infrastructure is achievable. This study is conducted in collaboration with Voi Technology, a Swedish micromobility company that is committed to eliminate all serious injuries and fatalities in their value chain by 2030. Given such an ambition, the aim of the thesis is to evaluate the possibility of using unsupervised machine learning for anomaly detection with sensor data, to distinguish abnormal and normal riding behaviours. The study evaluates two machine learning algorithms; isolation forest and artificial neural networks, namely autoencoders. Beyond assessing the models ability to detect abnormal riding behaviours in general, they are evaluated based on their ability to find certain behaviours. By simulating different abnormal riding behaviours, model evaluation can be performed. The data preparation performed for the models include transforming the time series data into non-overlapping windows of a specific size containing descriptive statistics. The result obtained shows that finding a one-size-fits all type of anomaly detection model did not work as desired for either the isolation forest or the autoencoder. Further, the result indicate that one of the abnormal riding behaviours appears to be easier to distinguish, which motivates evaluating models created with the aim of distinguishing that specific behaviour. Hence, a simple moving average is also implemented to explore the performance of a very basic forecasting method. For this method, a similar data transformation as previously described is not performed as it utilises a sliding window of specific size, which is run on a single feature corresponding to an entire scooter ride. The result show that it is possible to isolate one type of abnormal riding behaviour using the autoencoder model. Additionally, the simple moving average model can also be utilised to detect the behaviour in question. Out of the two models, it is recommended to deploy a simple moving average due to its simplicity. / Den globala mikromobilitetsmarknaden är en snabbt växande marknad som år 2020 värderades till 40,19 miljarder USD. I takt med att marknaden växer så ökar också kraven bland företag att erbjuda produkter och tjänster av hög kvalitet, för att  erhålla en stark position på marknaden, vara konkurrenskraftiga och förbli ett förstahandsval hos sina kunder. Detta kan uppnås genom att bland annat erbjuda mikromobilitetstjänster som är säkra, för både föraren och andra trafikanter. Med hjälp av den senaste tekniken kan olyckor förebyggas och skadligt bruk av skotrar och städers infrastruktur förhindras. Följande studie utförs i samarbete med Voi Technology, ett svenskt mikromobilitetsföretag som har åtagit sig ansvaret att eliminera samtliga allvarliga skador och dödsfall i deras värdekedja till och med år 2030. I linje med en sådan ambition, är syftet med avhandlingen att utvärdera möjligheten att använda oövervakad maskininlärning för anomalidetektering bland sensordata, för att särskilja onormala och normala körbeteenden. Studien utvärderar två maskininlärningsalgoritmer; isolation forest och artificiella neurala nätverk, mer specifikt autoencoders. Utöver att bedöma modellernas förmåga att upptäcka onormala körbeteenden i allmänhet, utvärderas modellerna utifrån deras förmåga att hitta särskilda körbeteenden. Genom att simulera olika onormala körbeteenden kan modellerna evalueras. Dataförberedelsen som utförs för modellerna inkluderar omvandling av den råa tidsseriedatan till icke överlappande fönster av specifik storlek, bestående av beskrivande statistik. Det erhållna resultatet visar att varken isolation forest eller autoencodern presterar som förväntat samt att önskan om att hitta en generell modell som klarar av att detektera anomalier av olika karaktär inte verkar uppfyllas. Vidare indikerar resultatet på att ett visst onormalt körbeteende verkar enklare att särskilja än resterande, vilket motiverar att utvärdera modeller skapade i syfte att detektera det specifika beteendet. Följaktligen implementeras därför ett glidande medelvärde för att utforska prestandan hos en mycket grundläggande prediktionsmetod. För denna metod utförs inte den tidigare nämnda datatransformationen eftersom metoden använder ett glidande medelvärde som appliceras på en variabel tillhörande en fullständig åktur.  Följande analys visar att autoencoder modellen klarar av att urskilja denna typ av onormalt körbeteende. Resultatet visar även att ett glidande medelvärde klarar av att detektera körbeteendet i fråga. Av de två modellerna rekommenderas en implementering av ett glidande medelvärdet på grund av dess enkelhet.