abstract: Image-based process monitoring has recently attracted increasing attention due to the advancement of the sensing technologies. However, existing process monitoring methods fail to fully utilize the spatial information of images due to their complex characteristics including the high dimensionality and complex spatial structures. Recent advancement of the unsupervised deep models such as a generative adversarial network (GAN) and generative adversarial autoencoder (AAE) has enabled to learn the complex spatial structures automatically. Inspired by this advancement, we propose an anomaly detection framework based on the AAE for unsupervised anomaly detection for images. AAE combines the power of GAN with the variational autoencoder, which serves as a nonlinear dimension reduction technique with regularization from the discriminator. Based on this, we propose a monitoring statistic efficiently capturing the change of the image data. The performance of the proposed AAE-based anomaly detection algorithm is validated through a simulation study and real case study for rolling defect detection. / Dissertation/Thesis / Masters Thesis Industrial Engineering 2019

Deep generative models are essential to Natural Language Processing (NLP) due to their outstanding ability to use unlabelled data, to incorporate abundant linguistic features, and to learn interpretable dependencies among data. As the structure becomes deeper and more complex, having an effective and efficient inference method becomes increasingly important. In this thesis, neural variational inference is applied to carry out inference for deep generative models. While traditional variational methods derive an analytic approximation for the intractable distributions over latent variables, here we construct an inference network conditioned on the discrete text input to provide the variational distribution. The powerful neural networks are able to approximate complicated non-linear distributions and grant the possibilities for more interesting and complicated generative models. Therefore, we develop the potential of neural variational inference and apply it to a variety of models for NLP with continuous or discrete latent variables. This thesis is divided into three parts. Part I introduces a <b>generic variational inference framework</b> for generative and conditional models of text. For continuous or discrete latent variables, we apply a continuous reparameterisation trick or the REINFORCE algorithm to build low-variance gradient estimators. To further explore Bayesian non-parametrics in deep neural networks, we propose a family of neural networks that parameterise categorical distributions with continuous latent variables. Using the stick-breaking construction, an unbounded categorical distribution is incorporated into our deep generative models which can be optimised by stochastic gradient back-propagation with a continuous reparameterisation. Part II explores <b>continuous latent variable models for NLP</b>. Chapter 3 discusses the Neural Variational Document Model (NVDM): an unsupervised generative model of text which aims to extract a continuous semantic latent variable for each document. In Chapter 4, the neural topic models modify the neural document models by parameterising categorical distributions with continuous latent variables, where the topics are explicitly modelled by discrete latent variables. The models are further extended to neural unbounded topic models with the help of stick-breaking construction, and a truncation-free variational inference method is proposed based on a Recurrent Stick-breaking construction (RSB). Chapter 5 describes the Neural Answer Selection Model (NASM) for learning a latent stochastic attention mechanism to model the semantics of question-answer pairs and predict their relatedness. Part III discusses <b>discrete latent variable models</b>. Chapter 6 introduces latent sentence compression models. The Auto-encoding Sentence Compression Model (ASC), as a discrete variational auto-encoder, generates a sentence by a sequence of discrete latent variables representing explicit words. The Forced Attention Sentence Compression Model (FSC) incorporates a combined pointer network biased towards the usage of words from source sentence, which significantly improves the performance when jointly trained with the ASC model in a semi-supervised learning fashion. Chapter 7 describes the Latent Intention Dialogue Models (LIDM) that employ a discrete latent variable to learn underlying dialogue intentions. Additionally, the latent intentions can be interpreted as actions guiding the generation of machine responses, which could be further refined autonomously by reinforcement learning. Finally, Chapter 8 summarizes our findings and directions for future work.

abstract: Feature learning and the discovery of nonlinear variation patterns in high-dimensional data is an important task in many problem domains, such as imaging, streaming data from sensors, and manufacturing. This dissertation presents several methods for learning and visualizing nonlinear variation in high-dimensional data. First, an automated method for discovering nonlinear variation patterns using deep learning autoencoders is proposed. The approach provides a functional mapping from a low-dimensional representation to the original spatially-dense data that is both interpretable and efficient with respect to preserving information. Experimental results indicate that deep learning autoencoders outperform manifold learning and principal component analysis in reproducing the original data from the learned variation sources. A key issue in using autoencoders for nonlinear variation pattern discovery is to encourage the learning of solutions where each feature represents a unique variation source, which we define as distinct features. This problem of learning distinct features is also referred to as disentangling factors of variation in the representation learning literature. The remainder of this dissertation highlights and provides solutions for this important problem. An alternating autoencoder training method is presented and a new measure motivated by orthogonal loadings in linear models is proposed to quantify feature distinctness in the nonlinear models. Simulated point cloud data and handwritten digit images illustrate that standard training methods for autoencoders consistently mix the true variation sources in the learned low-dimensional representation, whereas the alternating method produces solutions with more distinct patterns. Finally, a new regularization method for learning distinct nonlinear features using autoencoders is proposed. Motivated in-part by the properties of linear solutions, a series of learning constraints are implemented via regularization penalties during stochastic gradient descent training. These include the orthogonality of tangent vectors to the manifold, the correlation between learned features, and the distributions of the learned features. This regularized learning approach yields low-dimensional representations which can be better interpreted and used to identify the true sources of variation impacting a high-dimensional feature space. Experimental results demonstrate the effectiveness of this method for nonlinear variation pattern discovery on both simulated and real data sets. / Dissertation/Thesis / Doctoral Dissertation Industrial Engineering 2016

Indiana University-Purdue University Indianapolis (IUPUI) / Many complex processes can be viewed as sequential events on a network. In this thesis, we study the interplay between a network and the event sequences on it. We first focus on predicting events on a known network. Examples of such include: modeling retweet cascades, forecasting earthquakes, and tracing the source of a pandemic. In specific, given the network structure, we solve two types of problems - (1) forecasting future events based on the historical events, and (2) identifying the initial event(s) based on some later observations of the dynamics. The inverse problem of inferring the unknown network topology or links, based on the events, is also of great important. Examples along this line include: constructing influence networks among Twitter users from their tweets, soliciting new members to join an event based on their participation history, and recommending positions for job seekers according to their work experience. Following this direction, we study two types of problems - (1) recovering influence networks, and (2) predicting links between a node and a group of nodes, from event sequences.

Táto práca sa zameriava na analýzu vybraných častí GPON rámca pomocou algoritmov strojového učenia implementovaných pomocou knižnice TensorFlow. Vzhľadom na to, že GPON protokol je definovaný ako sada odporúčaní, implementácia naprieč spoločnosťami sa môže líšiť od navrhnutého protokolu. Preto analýza pomocou zásobníkového automatu nie je dostatočná. Hlavnou myšlienkou je vytvoriť systém modelov za použitia knižnice TensorFlow v Python3, ktoré sú schopné detekovať abnormality v komunikácií. Tieto modely používajú viaceré architektúry neuronových sietí (napr. LSTM, autoencoder) a zameriavajú sa na rôzne typy analýzy. Tento systém sa naučí na vzorovej vzorke dát a upozorní na nájdené odlišnosti v novozachytenej komunikácií. Výstupom systému odhad podobnosti aktuálnej komunikácie v porovnaní so vzorovou komunikáciou.

Magnetic resonance imaging (MRI) is a widely used medical imaging technique for examinations of the body. However, artifacts are a common problem, that must be handled for reliable diagnoses and to avoid drawing inaccurate conclusions about the contextual insights. Magnetic resonance (MR) images acquired with a Dixon-sequence enables two channels with separate fat and water content. Fat-water inversions, also called swaps, are one common artifact with this method where voxels from the two channels are swapped, producing incorrect data. This thesis investigates the possibility to use deep learning methods for an automatic detection of swaps in MR volumes. The data used in this thesis are MR volumes from UK Biobank, processed by AMRA Medical. Segmentation masks of complicated swaps are created by operators who manually annotate the swap, but only if the regions affect subsequent measurements. The segmentation masks are therefore not fully reliable, and additional synthesized swaps were created. Two different deep learning approaches were investigated, a reconstruction-based method and a segmentation-based method. The reconstruction-based networks were trained to reconstruct a volume as similar as possible to the input volume without any swaps. When testing the network on a volume with a swap, the location of the swap can be estimated from the reconstructed volume with postprocessing methods. Autoencoders is an example of a reconstruction-based network. The segmentation-based models were trained to segment a swap directly from the input volume, thus using volumes with swaps both during training and testing. The segmentation-based networks were inspired by a U-Net. The performance of the models from both approaches was evaluated on data with real and synthetic swaps with the metrics: Dice coefficient, precision, and recall. The result shows that the reconstruction-based models are not suitable for swap detection. Difficulties in finding the right architecture for the models resulted in bad reconstructions, giving unreliable predictions. Further investigations in different post-processing methods, architectures, and hyperparameters might improve swap detection. The segmentation-based models are robust with reliable detections independent of the size of the swaps, despite being trained on data with synthesized swaps. The results from the models look very promising, and can probably be used as an automated method for swap detection with some further fine-tuning of the parameters.

High performance and high bandwidth wireless digital communication underlies much of modern society. Due to its high value to society, new and improved digital communication technologies, allowing even higher speeds, better coverage, and lower latency are constantly being developed. The field of Machine Learning has exploded in recent years, showing incredible promise and performance at many tasks in a wide variety of fields. Channel Equalization and synchronization are critical parts of any wireless communication system, to ensure coherence between the transmitter and receiver, and to compensate for the often severe channel conditions. This study mainly explores the use of a Variational Autoencoder (VAE) architecture, presented in a previous study, for blind channel equalization without access to pilot symbols or ground-truth data. This thesis also presents a new, non data-assisted method of carrier frequency synchronization based around the k-means clustering algorithm. The main addition of this thesis however is a constellation agnostic implementation of the reference VAE architecture, for equalization of all rectangular QAM constellations. The approach significantly outperforms the traditional blind adaptive Constant Modulus algorithm (CMA) on all tested constellations and signal to noise ratios (SNRs), nearly equaling the performance of a non-blind Least Mean Squares (LMS) based Decision Feedback Equalizer (DFE).

This thesis focuses on modification of feature extraction and multiview object detection learning process. We add new channels to detectors based on the "Aggregate channel features" framework. These new channels are created by filtering the picture by kernels from autoencoders followed by nonlinear function processing. Experiments show that these channels are effective in detection but they are also more computationally expensive. The thesis therefore discusses possibilities for improvements. Finally the thesis evaluates an artificial car dataset and discusses its small benefit on several detectors.

Understanding travel behavior and identifying the mode of transportation are essential for adequate urban devising and transportation planning. Global positioning systems (GPS) tracking data is mainly used to find human mobility patterns in cities. Some travel information, such as most visited location, temporal changes, and the trip speed, can be easily extracted from GPS raw tracking data. GPS trajectories can be used as a method to indicate the mobility modes of commuters. Most previous studies have applied traditional machine learning algorithms and manually computed data features, making the model error-prone. Thus, there is a demand for developing a new model to resolve these methods' weaknesses. The primary purpose of this study is to propose a semi-supervised model to identify transportation mode by using a contemporary machine learning algorithm and GPS tracking data. The model can accept GPS trajectory with adjustable length and extracts their latent information with LSTM Autoencoder. This study adopts a deep neural network architecture with three hidden layers to map the latent information to detect transportation mode. Moreover, different case studies are performed to evaluate the proposed model's efficiency. The model results in an accuracy of 93.6%, which significantly outperforms similar studies.

Accelerated advancements in technology, the Internet of Things, and cloud computing have spurred an emergence of unstructured data that is contributing to rapid growth in data volumes. No human can manage to keep up with monitoring and analyzing these unbounded data streams and thus predictive and analytic tools are needed. By leveraging machine learning this data can be converted into insights which are enabling datadriven decisions that can drastically accelerate innovation, improve user experience, and drive operational efficiency. The purpose of this thesis is to design and implement a system for real-time outlier detection using unbounded data streams and machine learning. Traditionally, this is accomplished by using alarm-thresholds on important system metrics. Yet, a static threshold cannot account for changes in trends and seasonality, changes in the system, or an increased system load. Thus, the intention is to leverage machine learning to instead look for deviations in the behavior of the data not caused by natural changes but by malfunctions. The use-case driving the thesis forward is real-time outlier detection in a Content Delivery Network (CDN). The input data includes Http-error messages received by clients, and contextual information like region, cache domains, and error codes, to provide tailormade predictions accounting for the trends in the data. The outlier detection system consists of a data collection pipeline leveraging the technique of stream processing, a MiniBatchKMeans clustering model that provides online clustering of incoming data according to their similar characteristics, and an LSTM AutoEncoder that accounts for temporal nature of the data and detects outlier data points in the clusters. An important finding is that an outlier is defined as an abnormal amount of outlier data points all originating from the same cluster, not a single outlier data point. Thus, the alerting system will be implementing an outlier percentage threshold. The experimental results show that an outlier is detected within one minute from a cache break-down. This triggers an alert to the system owners, containing graphs of the clustered data to narrow down the search area of the cause to enable preventive action towards the prominent incident. Further results show that within 2 minutes from fixing the cause the system will provide feedback that the actions taken were successful. Considering the real-time requirements of the CDN environment, it is concluded that the short delay for detection is indeed real-time. Proving that machine learning is indeed able to detect outliers in unbounded data streams in a real-time manner. Further analysis shows that the system is more accurate during peakhours when more data is in circulation than during none peak-hours, despite the temporal LSTM layers. Presumably, an effect from the model needing to train on more data to better account for seasonality and trends. Future work necessary to put the outlier detection system in production thus includes more training to improve accuracy and correctness. Furthermore, one could consider implementing necessary functionality for a production environment and possibly adding enhancing features that can automatically avert incidents detected and handle the causes of them.