Many machine learning problems involve dealing with a large amount of high-dimensional data across diverse domains. In addition, annotating or labeling the data is expensive as it involves significant human effort. This dissertation explores a joint solution to both these problems by exploiting the property that high-dimensional data in real-world application domains often lies on a lower-dimensional structure, whose geometry can be modeled as a graph or manifold. In particular, we propose a set of novel manifold-alignment based approaches for transfer learning. The proposed approaches transfer knowledge across different domains by finding low-dimensional embeddings of the datasets to a common latent space, which simultaneously match corresponding instances while preserving local or global geometry of each input dataset. We develop a novel two-step transfer learning method called Procrustes alignment. Procrustes alignment first maps the datasets to low-dimensional latent spaces reflecting their intrinsic geometries and then removes the translational, rotational and scaling components from one set so that the optimal alignment between the two sets can be achieved. This approach can preserve either global geometry or local geometry depending on the dimensionality reduction approach used in the first step. We propose a general one-step manifold alignment framework called manifold projections that can find alignments, both across instances as well as across features, while preserving local domain geometry. We develop and mathematically analyze several extensions of this framework to more challenging situations, including (1) when no correspondences across domains are given; (2) when the global geometry of each input domain needs to be respected; (3) when label information rather than correspondence information is available. A final contribution of this thesis is the study of multiscale methods for manifold alignment. Multiscale alignment automatically generates alignment results at different levels by discovering the shared intrinsic multilevel structures of the given datasets, providing a common representation across all input datasets.

No description available.

Machine Translation (MT) is the task of mapping a source language to a target language. The recent introduction of neural MT (NMT) has shown promising results for high-resource language, however, poorly performing for low-resource language (LRL) settings. Furthermore, the vast majority of the 7, 000+ languages around the world do not have parallel data, creating a zero-resource language (ZRL) scenario. In this thesis, we present our approach to improving NMT for LRL and ZRL, leveraging a multilingual NMT modeling (M-NMT), an approach that allows building a single NMT to translate across multiple source and target languages. This thesis begins by i) analyzing the effectiveness of M-NMT for LRL and ZRL translation tasks, spanning two NMT modeling architectures (Recurrent and Transformer), ii) presents a self-learning approach for improving the zero-shot translation directions of ZRLs, iii) proposes a dynamic transfer-learning approach from a pre-trained (parent) model to a LRL (child) model by tailoring to the vocabulary entries of the latter, iv) extends M-NMT to translate from a source language to specific language varieties (e.g. dialects), and finally, v) proposes an approach that can control the verbosity of an NMT model output. Our experimental findings show the effectiveness of the proposed approaches in improving NMT of LRLs and ZRLs.

Task-oriented dialogue (ToD) systems need to interpret the user's input to understand the user's needs (intent) and corresponding relevant information (slots). This process is performed by a Natural Language Understanding (NLU) component, which maps the text utterance into a semantic frame representation, involving two subtasks: intent classification (text classification) and slot filling (sequence tagging). Typically, new domains and languages are regularly added to the system to support more functionalities. Collecting domain-specific data and performing fine-grained annotation of large amounts of data every time a new domain and language is introduced can be expensive. Thus, developing an NLU model that generalizes well across domains and languages with less labeled data (low-resource) is crucial and remains challenging. This thesis focuses on investigating transfer learning and data augmentation methods for low-resource NLU in ToD. Our first contribution is a study of the potential of non-conversational text as a source for transfer. Most transfer learning approaches assume labeled conversational data as the source task and adapt the NLU model to the target task. We show that leveraging similar tasks from non-conversational text improves performance on target slot filling tasks through multi-task learning in low-resource settings. Second, we propose a set of lightweight augmentation methods that apply data transformation on token and sentence levels through slot value substitution and syntactic manipulation. Despite its simplicity, the performance is comparable to deep learning-based augmentation models, and it is effective on six languages on NLU tasks. Third, we investigate the effectiveness of domain adaptive pre-training for zero-shot cross-lingual NLU. In terms of overall performance, continued pre-training in English is effective across languages. This result indicates that the domain knowledge learned in English is transferable to other languages. In addition to that, domain similarity is essential. We show that intermediate pre-training data that is more similar – in terms of data distribution – to the target dataset yields better performance.

Anwendung von maschinellen Lernverfahren (ML) in der Produktionstechnik, in Zeiten der Industrie 4.0, stark angestiegen. Insbesondere die Datenverfügbarkeit ist an dieser Stelle elementar und für die erfolgreiche Umsetzung einer ML-Applikation Voraussetzung. Falls für eine gegebene Problemstellung die Datenmenge oder -qualität nicht ausreichend ist, können Techniken, wie die Datenaugmentierung, der Einsatz von synthetischen Daten sowie das Transferlernen von ähnlichen Datensätzen Abhilfe schaffen. Innerhalb dieser Ausarbeitung wird das Konzept des Transferlernens im Bereich das Radial-Axial Ringwalzens (RAW) angewendet und am Beispiel der Zeitreihenprognose des Außendurchmessers über die Prozesszeit durchgeführt. Das Radial-Axial Ringwalzen ist ein warmumformendes Verfahren und dient der nahtlosen Ringherstellung.

Due to the increasing computing power and corresponding algorithms, the use of machine learning (ML) in production technology has risen sharply in the age of Industry 4.0. Data availability in particular is fundamental at this point and a prerequisite for the successful implementation of a ML application. If the quantity or quality of data is insufficient for a given problem, techniques such as data augmentation, the use of synthetic data and transfer learning of similar data sets can provide a remedy. In this paper, the concept of transfer learning is applied in the field of radial-axial ring rolling (rarr) and implemented using the example of time series prediction of the outer diameter over the process time. Radial-axial ring rolling is a hot forming process and is used for seamless ring production.

Indiana University-Purdue University Indianapolis (IUPUI) / Biomechanical Big Data is of great significance to precision health applications, among which we take special interest in Physical Activity Detection (PAD). In this study, we have performed extensive research on deep learning-based PAD from biomechanical big data, focusing on the challenges raised by the need for real-time edge inference. First, considering there are many places we can place the motion sensors, we have thoroughly compared and analyzed the location difference in terms of deep learning-based PAD performance. We have further compared the difference among six sensor channels (3-axis accelerometer and 3-axis gyroscope). Second, we have selected the optimal sensor and the optimal sensor channel, which can not only provide sensor usage suggestions but also enable ultra-lowpower application on the edge. Third, we have investigated innovative methods to minimize the training effort of the deep learning model, leveraging the transfer learning strategy. More specifically, we propose to pre-train a transferable deep learning model using the data from other subjects and then fine-tune the model using limited data from the target-user. In such a way, we have found that, for single-channel case, the transfer learning can effectively increase the deep model performance even when the fine-tuning effort is very small. This research, demonstrated by comprehensive experimental evaluation, has shown the potential of ultra-low-power PAD with minimized sensor stream, and minimized training effort. / 2023-06-01

No description available.

This thesis focuses on integrating learning paradigms that ‘share to learn,’ i.e., Multitask Learning (MTL), and ‘learn (how) to share,’ i.e., meta learning. MTL involves learning several tasks simultaneously within a shared network structure so that the tasks can mutually benefit each other’s learning. While meta learning, better known as ‘learning to learn,’ is an approach to reducing the amount of time and computation required to learn a novel task by leveraging on knowledge accumulated over the course of numerous training episodes of various tasks. The learning process in the human brain is innate and natural. Even before birth, it is capable of learning and memorizing. As a consequence, humans do not learn everything from scratch, and because they are naturally capable of effortlessly transferring their knowledge between tasks, they quickly learn new skills. Humans naturally tend to believe that similar tasks have (somewhat) similar solutions or approaches, so sharing knowledge from a previous activity makes it feasible to learn a new task quickly in a few tries. For instance, the skills acquired while learning to ride a bike are helpful when learning to ride a motorbike, which is, in turn, helpful when learning to drive a car. This natural learning process, which involves sharing information between tasks, has inspired a few research areas in Deep Learning (DL), such as transfer learning, MTL, meta learning, Lifelong Learning (LL), and many more, to create similar neurally-weighted algorithms. These information-sharing algorithms exploit the knowledge gained from one task to improve the performance of another related task. However, they vary in terms of what information they share, when to share, and why to share. This thesis focuses particularly on MTL and meta learning, and presents a comprehensive explanation of both the learning paradigms. A theoretical comparison of both algorithms demonstrates that the strengths of one can outweigh the constraints of the other. Therefore, this work aims to combine MTL and meta learning to attain the best of both worlds. The main contribution of this thesis is Multi-task Meta Learning (MTML), an integration of MTL and meta learning. As the gradient (or optimization) based metalearning follows an episodic approach to train a network, we propose multi-task learning episodes to train a MTML network in this work. The basic idea is to train a multi-task model using bi-level meta-optimization so that when a new task is added, it can learn in fewer steps and perform at least as good as traditional single-task learning on the new task. The MTML paradigm is demonstrated on two publicly available datasets – the NYU-v2 and the taskonomy dataset, for which four tasks are considered, i.e., semantic segmentation, depth estimation, surface normal estimation, and edge detection. This work presents a comparative empirical analysis of MTML to single-task and multi-task learning, where it is evident that MTML excels for most tasks. The future direction of this work includes developing efficient and autonomous MTL architectures by exploiting the concepts of meta learning. The main goal will be to create a task-adaptive MTL, where meta learning may learn to select layers (or features) from the shared structure for every task because not all tasks require the same highlevel, fine-grained features from the shared network. This can be seen as another way of combining MTL and meta learning. It will also introduce modular learning in the multi-task architecture. Furthermore, this work can be extended to include multi-modal multi-task learning, which will help to study the contributions of each input modality to various tasks.

This thesis leverages vibration-based unsupervised learning and deep transfer learning to reduce the manual labour involved in building algorithms that perform intelligent fault detection (IFD) on roller element bearings. A review of theory and literature in the field of IFD is presented, and challenges are discussed. An issue is then introduced; current machine learning models built for IFD show strong performance on a small subset of specific data, but do not generalize to a broader range of applications. Signal processing, machine learning, and transfer learning concepts are then explained and discussed. Time-frequency fingerprinting, as well as feature engineering, is used in conjunction with principal component analysis (PCA) to prepare vibration signals to be clustered by a gaussian mixture model (GMM). This process allows for the intelligent referral of data towards algorithms that have performed well on similar datasets and favours the re-use of domain-specific tasks. An algorithm is then proposed that promotes generalization in convolutional neural networks (CNNs) and simplifies the hyperparameter tuning process to allow machine learning models to be applied to a broader set of problems. The machine learning process is then automated as much as possible through meta learning and ensemble models: data similarity measurements are used to evaluate the data fit for transfer and propose training guidelines. Throughout the thesis, three open-source bearing fault datasets are used to test and validate the hypotheses. This thesis focuses on developing and adapting current deep learning models to succeed in challenging domains and real-world scenarios, while improving performance with unsupervised learning and transfer learning.