<p dir="ltr">Despite their astonishing capacity to fit data, neural networks have difficulties extrapolating beyond training data distribution. When the out-of-distribution prediction task is formalized as a counterfactual query on a causal model, the reason for their extrapolation failure is clear: neural networks learn spurious correlations in the training data rather than features that are causally related to the target label. This thesis proposes to perform a causal search over a known family of causal models to learn robust (maximally invariant) predictors for single- and multiple-environment extrapolation tasks.</p><p dir="ltr">First, I formalize the out-of-distribution task as a counterfactual query over a structural causal model. For single-environment extrapolation, I argue that symmetries of the input data are valuable for training neural networks that can extrapolate. I introduce Asymmetry learning, a new learning paradigm that is guided by the hypothesis that all (known) symmetries are mandatory even without evidence in training, unless the learner deems it inconsistent with the training data. Asymmetry learning performs a causal model search to find the simplest causal model defining a causal connection between the target labels and the symmetry transformations that affect the label. My experiments on a variety of out-of-distribution tasks on images and sequences show that proposed methods extrapolate much better than the standard neural networks.</p><p dir="ltr">Then, I consider multiple-environment out-of-distribution tasks in dynamical system forecasting that arise due to shifts in initial conditions or parameters of the dynamical system. I identify key OOD challenges in the existing deep learning and physics-informed machine learning (PIML) methods for these tasks. To mitigate these drawbacks, I combine meta-learning and causal structure discovery over a family of given structural causal models to learn the underlying dynamical system. In three simulated forecasting tasks, I show that the proposed approach is 2x to 28x more robust than the baselines.</p>

Physics Informed Machine Learning (PIML) has emerged as the forefront of research in scientific machine learning with the key motivation of systematically coupling machine learning (ML) methods with prior domain knowledge often available in the form of physics supervision. Uncertainty quantification (UQ) is an important goal in many scientific use-cases, where the obtaining reliable ML model predictions and accessing the potential risks associated with them is crucial. In this thesis, we propose novel methodologies in three key areas for improving uncertainty quantification for PIML. First, we propose to explicitly infuse the physics prior in the form of monotonicity constraints through architectural modifications in neural networks for quantifying uncertainty. Second, we demonstrate a more general framework for quantifying uncertainty with PIML that is compatible with generic forms of physics supervision such as PDEs and closed form equations. Lastly, we study the limitations of physics-based loss in the context of Physics-informed Neural Networks (PINNs), and develop an efficient sampling strategy to mitigate the failure modes. / Doctor of Philosophy / Owing to the success of deep learning in computer vision and natural language processing there is a growing interest of using deep learning in scientific applications. In scientific applications, knowledge is available in the form of closed form equations, partial differential equations, etc. along with labeled data. My work focuses on developing deep learning methods that integrate these forms of supervision. Especially, my work focuses on building methods that can quantify uncertainty in deep learning models, which is an important goal for high-stakes applications.

This dissertation aims to provide critical methodological advancements for sensor fusion and physics-informed machine learning in metal additive manufacturing (MAM) to assist practitioners in detecting quality control structural anomalies. In MAM, there is an urgent need to improve knowledge of the internal layer fusion process and geometric variation occurring during the directed energy deposition processes. A core challenge lies in the cyclic heating process, which results in various structural abnormalities and deficiencies, reducing the reproducibility of manufactured components. Structural abnormalities include microstructural heterogeneities, porosity, deformation and distortion, and residual stresses. Data-driven monitoring in MAM is needed to capture process variability, but challenges arise due to the inability to capture the thermal history distribution process and structural changes below the surface due to limitations in in-situ data collection capabilities, physical domain knowledge integration, and multi-data and multi-physical data fusion. The research gaps in developing system-based generalizable artificial intelligence (AI) and machine learning (ML) to detect abnormalities are threefold. (1) Limited fusion of various types of sensor data without handcrafted selection of features. (2) There is a lack of physical domain knowledge integration for various systems, geometries, and materials. (3) It is essential to develop sensor and system integration platforms to enable a holistic view to make quality control predictions in the additive manufacturing process. In this dissertation, three studies utilize various data types and ML methodologies for predicting in-process anomalies. First, a complementary sensor fusion methodology joins thermal and ultrasonic image data capturing layer fusion and structural knowledge for layer-wise porosity segmentation. Secondly, a physics-informed data-driven methodology for joining thermal infrared image data with Goldak heat flux improves thermal history simulation and deformation detection. Lastly, a physics-informed machine learning methodology constrained by thermal physical functions utilizes in-process multi-modal monitoring data from a digital twin environment to predict distortion in the weld bead. This dissertation provides current practitioners with data-driven and physics-based interpolation methods, multi-modal sensor fusion, and anomaly detection insights trained and validated with three case studies.

While machine learning has shown success in many fields, it can be challenging when there are limitations with insufficient training data. By incorporating knowledge into the machine learning pipeline, one can overcome such limitations. Therefore, eliciting expert knowledge can play an important role in the machine learning project pipeline. Expert knowledge can come in many forms, and it is seldom easy to elicit and formalize it in a way that is easily implementable into a machine learning project. While it has been done, not much focus has been on how. Furthermore, the motivations for why knowledge was elicited in a particular way as well as the challenges that may exist with the elicitation, are not always focused on either. Making educated decisions for knowledge elicitation can therefore be challenging for researchers. Hence, this work aims to explore and categorize how expert knowledge elicitation has been done by researchers previously. This was done by developing a taxonomy that was then used for analyzing articles. A total of 43 articles were found, containing 97 elicitation paths that were categorized in order to identify trends and common approaches. The findings from our study were used to provide guidance for an industrial case in its initial stage to show how the taxonomy presented in this work can be applied in a real-world scenario.

<p dir="ltr">Machine learning and deep learning techniques now provide innovative tools for addressing problems in biological, engineering, and physical systems. Physics-informed neural networks (PINNs) are a type of neural network that incorporate physical laws described by partial differential equations (PDEs) into their supervised learning tasks. This dissertation aims to enhance PINNs with improved training techniques and loss functions to tackle the complex physics of viscoelastic flow and rheology more effectively. The focus areas of the dissertation are listed as follows: i) Assigning relative weights to loss terms in training physics-informed neural networks (PINNs) is complex. We propose a solution using numerical integration via backward Euler discretization to leverage statistical properties of data for determining loss weights. Our study focuses on two and three-dimensional Navier-Stokes equations, using spatio-temporal velocity and pressure data to ascertain kinematic viscosity. We examine two-dimensional flow past a cylinder and three-dimensional flow within an aneurysm. Our method, tested for sensitivity and robustness against various factors, converges faster and more accurately than traditional PINNs, especially for three-dimensional Navier-Stokes equations. We validated our approach with experimental data, using the velocity field from PIV channel flow measurements to generate a reference pressure field and determine water viscosity at room temperature. Results showed strong performance with experimental datasets. Our proposed method is a promising solution for ’stiff’ PDEs and scenarios requiring numerous constraints where traditional PINNs struggle. ii) Machine learning algorithms are valuable for fluid mechanics, but high data costs limit their practicality. To address this, we present viscoelasticNet, a Physics-Informed Neural Network (PINN) framework that selects the appropriate viscoelastic constitutive model and learns the stress field from a given velocity flow field. We incorporate three non-linear viscoelastic models: Oldroyd-B, Giesekus, and Linear PTT. Our framework uses neural networks to represent velocity, pressure, and stress fields and employs the backward Euler method to construct PINNs for the viscoelastic model. The approach is multistage: first, it solves for stress, then uses stress and velocity fields to solve for pressure. ViscoelasticNet effectively learned the parameters of the viscoelastic constitutive model on noisy and sparse datasets. Applied to a two-dimensional stenosis geometry and cross-slot flow, our framework accurately learned constitutive equation parameters, though it struggled with peak stress at cross-slot corners. We suggest addressing this by exploring smaller domains. ViscoelasticNet can extend to other rheological models like FENE-P and extended Pom-Pom and learn entire equations, not just parameters. Future research could explore more complex geometries and three-dimensional cases. Complementing Particle Image Velocimetry (PIV), our method can determine pressure and stress fields once the constitutive equation is learned, allowing the modeling of future fluid applications. iii) Physics-Informed Neural Networks (PINNs) are widely used for solving inverse and forward problems in various scientific and engineering fields. However, most PINNs frameworks operate within the Eulerian domain, where physical quantities are described at fixed points in space. We explore coupling Eulerian and Lagrangian domains using PINNs. By tracking particles in the Lagrangian domain, we aim to learn the velocity field in the Eulerian domain. We begin with a sensitivity analysis, focusing on the time-step size of particle data and the number of particles. Initial tests with external flow past a cylinder show that smaller time-step sizes yield better results, while the number of particles has little effect on accuracy. We then extend our analysis to a real-world scenario: the interior of an airplane cabin. Here, we successfully reconstruct the velocity field by tracking passive particles. Our findings suggest that this coupled Eulerian-Lagrangian PINNs framework is a promising tool for enhancing traditional experimental techniques like particle tracking. It can be extended to learn additional flow properties, such as the pressure field for three-dimensional internal flows, and infer viscosity from passive particle tracking, providing deeper insights into complex fluids and their constitutive models. iv) Time-fractional differential equations are widely used across various fields but often present computational and stability challenges, especially in inverse problems. Leveraging Physics-Informed Neural Networks (PINNs) offers a promising solution for these issues. PINNs efficiently compute fractional time derivatives using finite differences and handle other derivatives via automatic differentiation. This study addresses two inverse problems: (1) anomalous diffusion and (2) fractional viscoelasticity. Our approach defines residual loss scaled with the standard deviation of observed data, using numerically generated and experimental datasets to learn fractional coefficients and calibrate parameters for the fractional Maxwell model. Our framework demonstrated robust performance for anomalous diffusion, maintaining less than 10% relative error in predicting the generalized diffusion coefficient and the fractional derivative order, even with 25% Gaussian noise added to the dataset. This highlights the framework’s resilience and accuracy in noisy conditions. We also validated our approach by predicting relaxation moduli for pig tissue samples, achieving relative errors below 10% compared to literature values. This underscores the efficacy of our fractional model with fewer parameters. Our method can be extended to model non-linear fractional viscoelasticity, incorporate experimental data for anomalous diffusion, and apply it to three-dimensional scenarios, broadening its practical applications.</p>

This study presents a procedure for the estimation of stopping behavior of industrial robots with a trained neural network. This trained network is presented as a single channel in a redundant architecture for safety control applications, where its potential for future integration with an analytical model of robot stopping is discussed. Basic physical relations for simplified articulated manipulators are derived, which motivate a choice of quantities to predict robot stopping behavior and inform the training and testing of a network for prediction of stopping distances and times. Robot stopping behavior is considered in the context of relevant standards ISO 10218-1, ISO/TS 15066 and IS0 13849-1, which inform the definitions for safety related stopping distances and times used in this study. Prior work on the estimation of robot stopping behavior is discussed alongside applications of machine learning to the broader field of industrial robotics, and particularly to the cases of prediction of forward and inverse kinematics with trained networks. A state-driven data collection program is developed to perform repeated stopping experiments for a controlled stop on path within a specified sampling domain. This program is used to collect data for a simulated and real robot system. Special attention is given to the identification of meaningful stopping times, which includes the separation of stopping into pre-deceleration and post-deceleration phases. A definition is provided for stopping of a robot in a safety context, based on the observation that residual motion over short distances (less than 1 mm) and at very low velocities (less than 1 mm/s) is not relevant to robot safety. A network architecture and hyperparameters are developed for the prediction of stopping distances and times for the first three joints of the manipulator without the inclusion of payloads. The result is a dual-network structure, where stopping distance predictions from the distance prediction network serve as inputs to the stopping time prediction network. The networks are validated on their capacity to interpolate and extrapolate predictions of robot stopping behavior in the presence of initial conditions not included in the training and testing data. A method is devised for the calculation of prediction errors for training training, testing and validation data. This method is applied both to interpolation and extrapolation to new initial velocity and positional conditions of the manipulator. In prediction of stopping distances and times, the network is highly successful at interpolation, resulting in comparable or nominally higher errors for the validation data set when compared to the errors for training and testing data. In extrapolation to new initial velocity and positional conditions, notably higher errors in the validation data predictions are observed for the networks considered. Future work in the areas of predictions of stopping behavior with payloads and tooling, further applications to collaborative robotics, analytical models of stopping behavior, inclusion of additional stopping functions, use of explainable AI methods and physics-informed networks are discussed. / Master of Science / As the uses for industrial robots continue to grow and expand, so do the need for robust safety measures to avoid, control, or limit the risks posed to human operators and collaborators. This is exemplified by Isaac Asimov's famous first law of robotics - "A robot may not injure a human being, or, through inaction, allow a human being to come to harm." As applications for industrial robots continue to expand, it is beneficial for robots and human operators to collaborate in work environments without fences. In order to ethically implement such increasingly complex and collaborative industrial robotic systems, the ability to limit robot motion with safety functions in a predictable and reliable way (as outlined by international standards) is paramount. In the event of either a technical failure (due to malfunction of sensors or mechanical hardware) or change in environmental conditions, it is important to be able to stop an industrial robot from any position in a safe and controlled manner. This requires real-time knowledge of the stopping distance and time for the manipulator. To understand stopping distances and times reliability, multiple independent methods can be used and compared to predict stopping behavior. The use of machine learning methods is of particular interest in this context due to their speed of processing and the potential for basis on real recorded data. In this study, we will attempt to evaluate the efficacy of machine learning algorithms to predict stopping behavior and assess their potential for implementation alongside analytical models. A reliable, multi-method approach for estimating stopping distances and times could also enable further methods for safety in collaborative robotics such as Speed and Separation Monitoring (SSM), which monitors both human and robot positions to ensure that a safe stop is always possible. A program for testing and recording the stopping distances and times for the robot is developed. As stopping behavior varies based on the positions and speeds of the robot at the time of stopping, a variety of these criteria are tested with the robot stopping program. This data is then used to train an artificial neural network, a machine learning method that mimics the structure of human and animal brains to learn relationships between data inputs and outputs. This network is used to predict both the stopping distance and time of the robot. The network is shown to produce reasonable predictions, especially for positions and speeds that are intermediate to those used to train the network. Future improvements are suggested and a method is suggested for use of stopping distance and time quantities in robot safety applications.

<p>The need to carry out Uncertainty quantification (UQ) is ubiquitous in science and engineering. However, carrying out UQ for real-world problems is not straightforward and they require a lot of computational budget and resources. The objective of this thesis is to develop computationally efficient approaches based on machine learning to carry out UQ. Specifically, we addressed two problems.</p> <p><br></p> <p>The first problem is, it is difficult to carry out Uncertainty propagation (UP) in systems governed by elliptic PDEs with spatially varying uncertain fields in coefficients and boundary conditions. Here as we have functional uncertainties, the number of uncertain parameters is large. Unfortunately, in these situations to carry out UP we need to solve the PDE a large number of times to obtain convergent statistics of the quantity governed by the PDE. However, solving the PDE by a numerical solver repeatedly leads to a computational burden. To address this we proposed to learn the surrogate of the solution of the PDE in a data-free manner by utilizing the physics available in the form of the PDE. We represented the solution of the PDE as a deep neural network parameterized function in space and uncertain parameters. We introduced a physics-informed loss function derived from variational principles to learn the parameters of the network. The accuracy of the learned surrogate is validated against the corresponding ground truth estimate from the numerical solver. We demonstrated the merit of using our approach by solving UP problems and inverse problems faster than by using a standard numerical solver.</p> <p><br></p> <p>The second problem we focused on in this thesis is related to inverse problems. State of the art approach to solving inverse problems involves posing the inverse problem as a Bayesian inference task and estimating the distribution of input parameters conditioned on the observed data (posterior). Markov Chain Monte Carlo (MCMC) methods and variational inference methods provides us ways to estimate the posterior. However, these inference techniques need to be re-run whenever a new set of observed data is given leading to a computational burden. To address this, we proposed to learn a Bayesian inverse map i.e., the map from the observed data to the posterior. This map enables us to do on-the-fly inference. We demonstrated our approach by solving various examples and we validated the posteriors learned from our approach against corresponding ground truth posteriors from the MCMC method.</p>

<p dir="ltr">In recent years, advancements in robotics and computing power have enabled robots to master complex tasks. Nevertheless, merely executing tasks isn't sufficient for robots. To achieve higher robot autonomy, learning the objective function is crucial. Autonomous systems can effectively eliminate the need for explicit programming by autonomously learning the control objective and deriving their control policy through the observation of task demonstrations. Hence, there's a need to develop a method for robots to learn the desired objective functions. In this thesis, we address several challenges in objective learning for autonomous systems, enhancing the applicability of our method in real-world scenarios. The ultimate objective of the thesis is to create a universal objective learning approach capable of addressing a range of existing challenges in the field while emphasizing data efficiency and robustness. Hence, building upon the previously mentioned intuition, we present a framework for autonomous systems to address a variety of objective learning tasks in real-time, even in the presence of noisy data. In addition to objective learning, this framework is capable of handling various other learning and control tasks.</p><p dir="ltr">The first part of this thesis concentrates on objective learning methods, specifically inverse optimal control (IOC). Within this domain, we have made three significant contributions aimed at addressing three existing challenges in IOC: 1) learning from minimal data, 2) learning without prior knowledge of system dynamics, and 3) learning with system outputs. </p><p dir="ltr">The second part of this thesis aims to develop a unified IOC framework to address all the challenges previously mentioned. It introduces a new paradigm for autonomous systems, referred to as Online Control-Informed Learning. This paradigm aims to tackle various of learning and control tasks online with data efficiency and robustness to noisy data. Integrating optimal control theory, online state estimation techniques, and machine learning methods, our proposed paradigm offers an online learning framework capable of tackling a diverse array of learning and control tasks. These include online imitation learning, online system identification, and policy tuning on-the-fly, all with efficient use of data and computation resources while ensuring robust performance.</p>

<p dir="ltr">Evidence for climate change-related impacts and risks is already widespread globally, affecting not only the ecosystems but also the economy and health of our communities. Data-driven predictive modeling approaches such as machine learning and deep learning have emerged to be powerful tools for interpreting large and complex non-linear datasets such as meteorological variables from weather stations or the distribution of infectious droplets produced in a cough. However, the strength of these data-driven models can be further optimized by complementing them with foundational knowledge of the physical processes they represent. By understanding the core physics, one can enhance the reliability and accuracy of predictive outcomes. The effectiveness of these combined approaches becomes particularly feasible and robust with the recent advancements in the High-Performance Computing field. With improved processing speed, algorithm design, and storage capabilities, modern computers allow for a deeper and more precise examination of the data. Such advancements equip us to address the diverse challenges presented by climate change more effectively.</p><p dir="ltr">In particular, this document advances research in mitigating and preventing the consequences of global warming by implementing data-driven predictive models based on statistical, machine learning, and deep learning methods via two phases. In the first phase, this dissertation proposes frameworks consisting of machine and deep learning algorithms to increase the resilience of small-scale renewable energy systems, which are essential for reducing greenhouse gas emissions in the ecosystems. The second phase focuses on using data from physics-based models, i.e., computational fluid dynamics (CFD), in data-driven predictive models for improving the design of air cleaning technologies, which are crucial to reducing the transmission of infectious diseases in indoor environments. </p><p dir="ltr">Specifically, this work is an article-based collection of published (or will be published) research articles. The articles are reformatted to fit the thesis's structure. The contents of the original articles are self-contained. </p>