<div>In this dissertation, we study two important problems in the area of modern deep learning: adversarial robustness and adversarial generative model. In the first part, we study the generalization performance of deep neural networks (DNNs) in adversarial learning. Recent studies have shown that many machine learning models are vulnerable to adversarial attacks, but much remains unknown concerning its generalization error in this scenario. We focus on the $\ell_\infty$ adversarial attacks produced under the fast gradient sign method (FGSM). We establish a tight bound for the adversarial Rademacher complexity of DNNs based on both spectral norms and ranks of weight matrices. The spectral norm and rank constraints imply that this class of networks can be realized as a subset of the class of a shallow network composed with a low dimensional Lipschitz continuous function. This crucial observation leads to a bound that improves the dependence on the network width compared to previous works and achieves depth independence. We show that adversarial Rademacher complexity is always larger than its natural counterpart, but the effect of adversarial perturbations can be limited under our weight normalization framework. </div><div></div><div>In the second part, we study deep generative models that receive great success in many fields. It is well-known that the complex data usually does not populate its ambient Euclidean space but resides in a lower-dimensional manifold instead. Thus, misspecifying the latent dimension in generative models will result in a mismatch of latent representations and poor generative qualities. To address these problems, we propose a novel framework called Latent Wasserstein GAN (LWGAN) to fuse the auto-encoder and WGAN such that the intrinsic dimension of data manifold can be adaptively learned by an informative latent distribution. In particular, we show that there exist an encoder network and a generator network in such a way that the intrinsic dimension of the learned encodes distribution is equal to the dimension of the data manifold. Theoretically, we prove the consistency of the estimation for the intrinsic dimension of the data manifold and derive a generalization error bound for LWGAN. Comprehensive empirical experiments verify our framework and show that LWGAN is able to identify the correct intrinsic dimension under several scenarios, and simultaneously generate high-quality synthetic data by samples from the learned latent distribution. </div><div><br></div>

Computational ECG (electrocardiogram) analysis enables accurate and faster diagnosis and early prediction of heart failure related symptoms (heart failure worsening). Machine learning, particularly deep learning, has been applied for ECG data successfully. The previous applications, however, either mainly focused on classifying occurrent, known patterns of on-going heart failure or heart failure related diseases such arrhythmia, which have undesirable predictability beforehand, or emphasizing on data from pre-processed public database data. In this dissertation, we developed an approach, however, does not fully capitalize on the potential of deep learning, which directly learns important features from raw input data without relying on a priori knowledge. Here, we present a deep transfer learning pipeline which combines an image-based pretrained deep neural network model with manifold learning to predict the precursors of heart failure (heart failure-worsening and recurrent heart failure related re-hospitalization) using raw ECG time series from wearable devices. In this dissertation, we used the unprocessed real-life ECG data from the SENTINEL-HF study by Dovancescu, et al. to predict the precursors of heart failure worsening. To extract rich features from ECG time series, we took a deep transfer learning approach where 1D time-series of five heartbeats were transformed to 2D images by Gramian Angular Summation Field (GASF) and then the pretrained models, VGG19 were used for feature extraction. Then, we applied UMAP (Uniform Manifold Approximation and Projection) to capture the manifold of the standardized feature space and reduce the dimension, followed by SVM (Support Vector Machine) training. Using our pipeline, we demonstrated that our classifier was able to predict heart failure worsening with 92.1% accuracy, 92.9% precision, 92.6% recall and F1 score of 0.93 bypassing the detection of known abnormal ECG patterns. In conclusion, we demonstrate the feasibility of early alerts of heart failure by predicting the precursor of heart failure worsening based on raw ECG signals. We expected that our approached provided an innovative method to assess the recovery and successfulness for the treatment patient received during the first hospitalization, to predict whether recurrent heart failure is likely to occur, and to evaluate whether the patient should be discharged.

High-dimensional data sets can be difficult to visualize and analyze, while data in low-dimensional space tend to be more accessible. In order to aid visualization of the underlying structure of a dataset, the dimension of the dataset is reduced. The simplest approach to accomplish this task of dimensionality reduction is by a random projection of the data. Even though this approach allows some degree of visualization of the underlying structure, it is possible to lose more interesting underlying structure within the data. In order to address this concern, various supervised and unsupervised linear dimensionality reduction algorithms have been designed, such as Principal Component Analysis and Linear Discriminant Analysis. These methods can be powerful, but often miss important non-linear structure in the data. In this thesis, manifold learning approaches to dimensionality reduction are developed. These approaches combine both linear and non-linear methods of dimension reduction.

Data that is represented with high dimensionality presents a computational complexity challenge for many existing algorithms. Limiting dimensionality by discarding attributes is sometimes a poor solution to this problem because significant high-level concepts may be encoded in the data across many or all of the attributes. Non-linear dimensionality reduction (NLDR) techniques have been successful with many problems at minimizing dimensionality while preserving intrinsic high-level concepts that are encoded with varying combinations of attributes. Unfortunately, many challenges remain with existing NLDR techniques, including excessive computational requirements, an inability to benefit from prior knowledge, and an inability to handle certain difficult conditions that occur in data with many real-world problems. Further, certain practical factors have limited advancement in NLDR, such as a lack of clarity regarding suitable applications for NLDR, and a general inavailability of efficient implementations of complex algorithms. This dissertation presents a collection of papers that advance the state of NLDR in each of these areas. Contributions of this dissertation include: • An NLDR algorithm, called Manifold Sculpting, that optimizes its solution using graduated optimization. This approach enables it to obtain better results than methods that only optimize an approximate problem. Additionally, Manifold Sculpting can benefit from prior knowledge about the problem. • An intelligent neighbor-finding technique called SAFFRON that improves the breadth of problems that existing NLDR techniques can handle. • A neighborhood refinement technique called CycleCut that further increases the robustness of existing NLDR techniques, and that can work in conjunction with SAFFRON to solve difficult problems. • Demonstrations of specific applications for NLDR techniques, including the estimation of state within dynamical systems, training of recurrent neural networks, and imputing missing values in data. • An open source toolkit containing each of the techniques described in this dissertation, as well as several existing NLDR algorithms, and other useful machine learning methods.

<p>In this dissertation, we study three important problems in modern deep learning: adversarial robustness, visualization, and partially monotonic function modeling. In the first part, we study the trade-off between robustness and standard accuracy in deep neural network (DNN) classifiers. We introduce sensible adversarial learning and demonstrate the synergistic effect between pursuits of standard natural accuracy and robustness. Specifically, we define a sensible adversary which is useful for learning a robust model while keeping high natural accuracy. We theoretically establish that the Bayes classifier is the most robust multi-class classifier with the 0-1 loss under sensible adversarial learning. We propose a novel and efficient algorithm that trains a robust model using implicit loss truncation. Our  experiments demonstrate that our method is effective in promoting robustness against various attacks and keeping high natural accuracy. </p> <p>In the second part, we study nonlinear dimensional reduction with the manifold assumption, often called manifold learning. Despite the recent advances in manifold learning, current state-of-the-art techniques focus on preserving only local or global structure information of the data. Moreover, they are transductive; the dimensional reduction results cannot be generalized to unseen data. We propose iGLoMAP, a novel inductive manifold learning method for dimensional reduction and high-dimensional data visualization. iGLoMAP preserves both local and global structure information in the same algorithm by preserving geodesic distance between data points. We establish the consistency property of our geodesic distance estimators. iGLoMAP can provide the lower-dimensional embedding for an unseen, novel point without any additional optimization. We  successfully apply iGLoMAP to the simulated and real-data settings with competitive experiments against state-of-the-art methods.</p> <p>In the third part, we study partially monotonic DNNs. We model such a function by using the fundamental theorem for line integrals, where the gradient is parametrized by DNNs. For the validity of the model formulation, we develop a symmetric penalty for gradient modeling. Unlike existing methods, our method allows partially monotonic modeling for general DNN architectures and monotonic constraints on multiple variables. We empirically show the necessity of the symmetric penalty on a simulated dataset.</p>

No description available.

Diese Dissertation beschäftigt sich mit zwei der wesentlichen Herausforderungen, welche bei der Bearbeitung großer Datensätze auftreten, der Dimensionsreduktion und der Datenentstörung. Der erste Teil dieser Dissertation liefert eine Zusammenfassung über Dimensionsreduktion. Ziel der Dimensionsreduktion ist eine sinnvolle niedrigdimensionale Darstellung eines vorliegenden hochdimensionalen Datensatzes. Insbesondere diskutieren und vergleichen wir bewährte Methoden des Manifold-Learning. Die zentrale Annahme des Manifold-Learning ist, dass der hochdimensionale Datensatz (approximativ) auf einer niedrigdimensionalen Mannigfaltigkeit liegt. Störungen im Datensatz sind bei allen Dimensionsreduktionsmethoden hinderlich. Der zweite Teil dieser Dissertation stellt eine neue Entstörungsmethode für hochdimensionale Daten vor, eine Wavelet-Shrinkage-Methode für die Glättung verrauschter Abtastwerte einer zugrundeliegenden multivariaten stückweise stetigen Funktion, wobei die Abtastpunkte gestreut sein können. Die Methode stellt eine Verallgemeinerung und Weiterentwicklung der für die Bildkompression eingeführten "Easy Path Wavelet Transform" (EPWT) dar. Grundlage ist eine eindimensionale Wavelet-Transformation entlang (adaptiv) zu konstruierender Pfade durch die Abtastpunkte. Wesentlich für den Erfolg der Methode sind passende adaptive Pfadkonstruktionen. Diese Dissertation beinhaltet weiterhin eine kurze Diskussion der theoretischen Eigenschaften von Wavelets entlang von Pfaden sowie numerische Resultate und schließt mit möglichen Modifikationen der Entstörungsmethode.

This thesis aims to contribute to the area of kernel methods, which are a class of machine learning methods known for their wide applicability and state-of-the-art performance, but which suffer from high training and evaluation complexity. The work in this thesis utilizes the notion of reduced-set models to alleviate the training and testing complexities of these methods in a unified manner. In the first part of the thesis, we use recent results in kernel smoothing and integral-operator learning to design a generic strategy to speed up various kernel methods. In Chapter 3, we present a method to speed up kernel PCA (KPCA), which is one of the fundamental kernel methods for manifold learning, by using reduced-set density estimates (RSDE) of the data. The proposed method induces an integral operator that is an approximation of the ideal integral operator associated to KPCA. It is shown that the error between the ideal and approximate integral operators is related to the error between the ideal and approximate kernel density estimates of the data. In Chapter 4, we derive similar approximation algorithms for Gaussian process regression, diffusion maps, and kernel embeddings of conditional distributions. In the second part of the thesis, we use reduced-set models for kernel methods to tackle online learning in model-reference adaptive control (MRAC). In Chapter 5, we relate the properties of the feature spaces induced by Mercer kernels to make a connection between persistency-of-excitation and the budgeted placement of kernels to minimize tracking and modeling error. In Chapter 6, we use a Gaussian process (GP) formulation of the modeling error to accommodate a larger class of errors, and design a reduced-set algorithm to learn a GP model of the modeling error. Proofs of stability for all the algorithms are presented, and simulation results on a challenging control problem validate the methods.

A patient-specific model of the liver can supply accurate volume measurements for oncologists and lesion locations and liver visualisation for surgeons. Our work seeks to enable an automatic computational tool for liver quantification. To create this model, the liver shape must be segmented from 3D CT images. In doing so, we can quantify liver volume and restrict the region of interest to ease the task of tumour and vascular segmentation. The main objective of liver segmentation developed into a mission to fluently describe liver shape a priori in level-set methods. This thesis looks at the utility of an implicit shape representation based on the Poisson equation to describe highly variable shapes, with application to image segmentation. Our first contribution is analyses on four implicit shape representations based on the heat equation, the signed distance function, Poisson’s equation, and the logarithm of odds. For four separate shape case studies, we summarise the class of shapes through their shape representation using Principal Component Analysis (PCA). Each shape class is highly variable across a population, but have a characteristic structure. We quantitatively compare the implicit shape representations, within each class, by evaluating its compactness, and in the last case, also completeness. To the best of our knowledge, this study is novel in comparing several shape representations through a single dimension reduction method. Our second contribution is a hybrid region-based level set segmentation that simultaneously infers liver shape given the image data, integrates the Poisson-based shape function prior into the segmentation, and evolves the level set according to the image data. We test our algorithm on exemplary 2D liver axial slices. We compare results for each image to results from (a) level-set segmentation without a shape prior and (b) level-set segmentation with a shape prior based on the Signed Distance Transform (SDT). In both priors, shapes are projected from shape space through the sample population mean and its modes of variation (the minimum number of principal components to comprise at least 95% of the cumulative variance). We compare results on four individual cases using the Dice coefficient and the Hausdorff distance. This thesis introduces an implicit shape representation based on Poisson’s equation in the field of medical image segmentation, showing its influence on shape space summary and projection. We analyse the shape space for compactness, showing that it is more compact in each of our case studies by at least two-fold and as much as three-fold. For 3D liver shapes, we show that it is more complete than the other three implicit shape representations. We utilise its description efficiency for use in 2D liver image segmentation, implementing the first shape function prior based on the Poisson equation. We show a qualitative and quantitative improvement over segmentation results without any shape prior and comparable results to segmentation with a SDT shape prior.

In this thesis, I look at three problems with important applications in data processing. Incorporating side information, provided by the user or derived from data, is a main theme of each of these problems. <br /><br /> This thesis makes a number of contributions. The first is a technique for combining different embedding objectives, which is then exploited to incorporate side information expressed in terms of transformation invariants known to hold in the data. It also introduces two different ways of incorporating transformation invariants in order to make new similarity measures. Two algorithms are proposed which learn metrics based on different types of side information. These learned metrics can then be used in subsequent embedding methods. Finally, it introduces a manifold learning algorithm that is useful when applied to sequential decision problems. In this case we are given action labels in addition to data points. Actions in the manifold learned by this algorithm have meaningful representations in that they are represented as simple transformations.