Deep learning has gained traction in thermal engineering due to its applications to process simulations, the deeper insights it can provide and its abilities to circumvent the shortcomings of classic thermodynamic simulation approaches by capturing complex inter-dependencies. This works sets out to apply probabilistic deep learning to power plant operations using historic plant data. The first study presented, entails the development of a steady-state mixture density network (MDN) capable of predicting effective heat transfer coefficients (HTC) for the various heat exchanger components inside a utility scale boiler. Selected directly controllable input features, including the excess air ratio, steam temperatures, flow rates and pressures are used to predict the HTCs. In the second case study, an encoder-decoder mixturedensity network (MDN) is developed using recurrent neural networks (RNN) for the prediction of utility-scale air-cooled condenser (ACC) backpressure. The effects of ambient conditions and plant operating parameters, such as extraction flow rate, on ACC performance is investigated. In both case studies, hyperparameter searches are done to determine the best performing architectures for these models. Comparisons are drawn between the MDN model versus standard model architecture in both case studies. The HTC predictor model achieved 90% accuracy which equates to an average error of 4.89 W m2K across all heat exchangers. The resultant time-series ACC model achieved an average error of 3.14 kPa, which translate into a model accuracy of 82%.

<p>Every engineered system behaves slightly differently because of manufacturing and operational uncertainties. The ability to build system-specific predictive models that adapt to manufactured systems, also known as digital twins, opens up many possibilities for reducing operating and maintenance costs. Nonlinear dynamical systems with unknown governing equations and states characterize many engineered systems. As a result, learning their dynamics from data has become both the current research area and one of the biggest challenges. In this thesis, we do an investigation of different data driven approaches for modeling various engineered systems. Firstly, we develop a model to predict the transient and steady-state behavior of a turbocharger turbine using the Koopman operator which can be helpful for modelling, analysis and control design. Our approach is as follows. We use experimental data from a Cummins heavy-duty diesel engine to develop a turbine model using Extended Dynamic Mode Decomposition (EDMD), which approximates the action of the Koopman operator on a finite-dimensional subspace of the space of observables. The results demonstrate comparable performance with a tuned nonlinear autoregressive network with an exogenous input (NARX) model widely used in the literature. The performance of these two models is analyzed based on their ability to predict turbine transient and steady-state behavior. Furthermore, we assess the ability of liquid time-constant (LTC) networks to learn the dynamics of various oscillatory systems using noisy data. In this study, we analyze and compare the performance of the LTC network with various commonly used recurrent neural network (RNN) architectures like long short-term memory (LSTM) network, and gated recurrent units (GRU). Our approach is as follows. We first systematically generate synthetic data by exciting the system of interest with a band-limited white noise and simulating it using a forward Euler discretization scheme. After the output has been simulated, we then corrupt it with different levels of noise to replicate a practically measured signal and train the RNN architectures with that corrupted output. The model is then tested on various types of forcing excitations to analyze the robustness of these networks in capturing different behaviors exhibited by the system. We also analyze the ability of these networks to capture the resonance effect for various parameter settings. Case studies discussing standard benchmark oscillatory systems (i.e., spring-mass-damper (S-M-D) system, single degree of freedom (DOF) Bouc-Wen oscillator, and forced Van der pol oscillator) are used to test the performance of these methodologies. The results reveal that the LTC network showed better performance in modeling the S-M-D system and 1-DOF Bouc-Wen oscillator as compared to an LSTM network but was outperformed by the GRU network. None of the networks were able to model the forced Van der pol oscillator with a reasonable accuracy. Since the GRU network outperformed other networks in terms of the computational time and the model accuracy for most of the scenarios, we applied it to a real world experimental dataset (i.e. turbocharger turbine dynamics) to compare it against the EDMD and NARX model. The results showed better performance of the GRU network in modeling the transient behaviours of the turbine. However, it failed to predict the turbine outlet temperature with a reasonable accuracy in most of the regions for the steady state dataset. As future work, we plan to consider training the GRU network with a data sampling frequency of 100 Hz for a fair comparison with the NARX and the Koopman approach.</p>

Spatiotemporal Predictive Learning (SPL) is an essential research topic involving many practical and real-world applications, e.g., motion detection, video generation, precipitation forecasting, and traffic flow prediction. The problems and challenges of this field come from numerous data characteristics in both time and space domains, and they vary depending on the specific task. For instance, spatial analysis refers to the study of spatial features, such as spatial location, latitude, elevation, longitude, the shape of objects, and other patterns. From the time domain perspective, the temporal analysis generally illustrates the time steps and time intervals of data points in the sequence, also known as interval recording or time sampling. Typically, there are two types of time sampling in temporal analysis: regular time sampling (i.e., the time interval is assumed to be fixed) and the irregular time sampling (i.e., the time interval is considered arbitrary) related closely to the continuous-time prediction task when data are in continuous space. Therefore, an efficient spatiotemporal predictive method has to model spatial features properly at the given time sampling types. In this thesis, by taking advantage of Machine Learning (ML) and Deep Learning (DL) methods, which have achieved promising performance in many complicated computational tasks, we propose three DL-based models used for Spatiotemporal Sequence Prediction (SSP) with several types of time sampling. First, we design the Trajectory Gated Recurrent Unit Attention (TrajGRU-Attention) with novel attention mechanisms, namely Motion-based Attention (MA), to improve the performance of the standard Convolutional Recurrent Neural Networks (ConvRNNs) in the SSP tasks. In particular, the TrajGRU-Attention model can alleviate the impact of the vanishing gradient, which leads to the blurry effect in the long-term predictions and handle both regularly sampled and irregularly sampled time series. Consequently, this model can work effectively with different scenarios of spatiotemporal sequential data, especially in the case of time series with missing time steps. Second, by taking the idea of Neural Ordinary Differential Equations (NODEs), we propose Trajectory Gated Recurrent Unit integrating Ordinary Differential Equation techniques (TrajGRU-ODE) as a continuous time-series model. With Ordinary Differential Equation (ODE) techniques and the TrajGRU neural network, this model can perform continuous-time spatiotemporal prediction tasks and generate resulting output with high accuracy. Compared to TrajGRU-Attention, TrajGRU-ODE benefits from the development of efficient and accurate ODE solvers. Ultimately, we attempt to combine those two models to create TrajGRU-Attention-ODE. NODEs are still in their early stage of research, and recent ODE-based models were designed for many relatively simple tasks. In this thesis, we will train the models with several video datasets to verify the ability of the proposed models in practical applications. To evaluate the performance of the proposed models, we select four available spatiotemporal datasets based on the complexity level, including the MovingMNIST, MovingMNIST++, and two real-life datasets: the weather radar HKO-7 and KTH Action. With each dataset, we train, validate, and test with distinct types of time sampling to justify the prediction ability of our models. In summary, the experimental results on the four datasets indicate the proposed models can generate predictions properly with high accuracy and sharpness. Significantly, the proposed models outperform state-of-the-art ODE-based approaches under SSP tasks with different circumstances of interval recording.

Due to the growing share of renewable energy in countries' power systems, the need for precise forecasting of electricity consumption will increase. This paper considers two different approaches to time series forecasting, autoregressive moving average (ARMA) models and recurrent neural networks (RNNs). These are applied to Swedish electricity consumption data, with the aim of deriving simple yet efficient predictors. An additional aim is to analyse the impact of day of week and temperature on forecast accuracy. The models are evaluated on both long- and mid-term forecasting horizons, ranging from one day to one month. The results show that neural networks are superior for this task, although stochastic seasonal ARMA models also perform quite well. Including external variables only marginally improved the ARMA predictions, and had somewhat unclear effects on the RNN forecasting accuracy. Depending on the network model used, adding external variables had either a slightly positive or slightly negative impact on prediction accuracy.

No description available.

Generative models are broadly used in many subfields of DL. DNNs have recently developed a core approach to solving data-centric problems in image classification, translation, etc. The latest developments in parameterizing these models using DNNs and stochastic optimization algorithms have allowed scalable modeling of complex, high-dimensional data, including speech, text, and image. This dissertation proposal presents our state-the-art probabilistic bases and DL algorithms for generative models, including VAEs, GANs, and RNN-based encoder-decoder. The proposal also discusses application areas that may benefit from deep generative models in both NLP and computer vision. In NLP, we proposed an Arabic poetry generation model with extended phonetic and semantic embeddings (Phonetic CNN_subword embeddings). Extensive quantitative experiments using BLEU scores and Hamming distance show notable enhancements over strong baselines. Additionally, a comprehensive human evaluation confirms that the poems generated by our model outperform the base models in criteria including meaning, coherence, fluency, and poeticness. We proposed a generative video model using a hybrid VAE-GAN model in computer vision. Besides, we integrate two attentional mechanisms with GAN to get the essential regions of interest in a video, focused on enhancing the visual implementation of the human motion in the generated output. We have considered quantitative and qualitative experiments, including comparisons with other state-of-the-arts for evaluation. Our results indicate that our model enhances performance compared with other models and performs favorably under different quantitive metrics PSNR, SSIM, LPIPS, and FVD.Recently, mimicking biologically inspired learning in generative models based on SNNs has been shown their effectiveness in different applications. SNNs are the third generation of neural networks, in which neurons communicate through binary signals known as spikes. Since SNNs are more energy-efficient than DNNs. Moreover, DNN models have been vulnerable to small adversarial perturbations that cause misclassification of legitimate images. This dissertation shows the proposed ``VAE-Sleep'' that combines ideas from VAE and the sleep mechanism leveraging the advantages of deep and spiking neural networks (DNN--SNN).On top of that, we present ``Defense–VAE–Sleep'' that extended work of ``VAE-Sleep'' model used to purge adversarial perturbations from contaminated images. We demonstrate the benefit of sleep in improving the generalization performance of the traditional VAE when the testing data differ in specific ways even by a small amount from the training data. We conduct extensive experiments, including comparisons with the state–of–the–art on different datasets.

The growing interest in more electrified vehicles has been pushing the industry and academia to pursue new and more accurate ways to estimate the xEV batteries State-of-Charge (SOC). The battery system still represents one of the many technical barriers that need to be eliminated or reduced to enable the proliferation of more xEV in the market, which in turn can help reduce CO2 emissions. Battery modelling and SOC estimation of Lithium-ion batteries (Li-ion) at a wide temperature range, including negative temperatures, has been a challenge for many engineers. For SOC estimation, several models configurations and approaches were developed and tested as results of this work, including different non-recurrent neural networks, such as Feedforward deep neural networks (FNN) and recurrent neural networks based on long short-term memory recurrent neural networks (LSTM-RNN). The approaches have considerably improved the accuracy presented in the previous state-of-the-art. They have expanded the application throughout five different Li-ion at a wide temperature range, achieving error as low as 0.66% Root Mean Square Error at -10⁰C using an FNN approach and 0.90% using LSTM-RNN. Therefore, the use of deep neural networks developed in this work can increase the potential for xEV application, especially where accuracy at negative temperatures is essential. For Li-ion modelling, a cell model using LSTM-RNN (LSTM-VM) was developed for the first time to estimate the battery cell terminal voltage and is compared against a gated recurrent unit (GRU-VM) approach and a Third-order Equivalent Circuit Model based on Thevenin theorem (ECM). The models were extensively compared for different Li-ion at a wide range of temperature conditions. The LSTM-VM has shown to be more accurate than the two other benchmarks, where could achieve 43 (mV) Root Mean Square Error at -20⁰C, a third when compared to the same situation using ECM. Although the difference between LSTM-VM and GRU-VM is not that steep. Finally, throughout the work, several methods to improve robustness, accuracy and training time have been introduced, including Transfer Learning applied to the development of SOC estimation models, showing great potential to reduce the amount of data necessary to train LSTM-RNN as well as improve its accuracy. / Thesis / Doctor of Philosophy (PhD) / For electric vehicle State-of-Charge estimation, several models configurations and approaches were developed and tested as results of this work, including different non-recurrent neural networks, such as Feedforward deep neural networks (FNN) and recurrent neural networks based on long short-term memory recurrent neural networks (LSTM-RNN). The approaches have considerably improved the accuracy presented in the previous state-of-the-art. They have expanded the application throughout five different Li-ion at a wide temperature range, achieving error as low as 0.66% Root Mean Square Error at -10⁰C using an FNN approach and 0.90% using LSTM-RNN. Therefore, the use of deep neural networks developed in this work can increase the potential for xEV application, especially where accuracy at negative temperatures is essential. For Li-ion modelling, a cell model using LSTM-RNN (LSTM-VM) was developed for the first time to estimate the battery cell terminal voltage and is compared against a gated recurrent unit (GRU-VM) approach and a Third-order Equivalent Circuit Model based on Thevenin theorem (ECM). The models were extensively compared for different Li-ion at a wide range of temperature conditions. The LSTM-VM has shown to be more accurate than the two other benchmarks, where could achieve 43 (mV) Root Mean Square Error at -20⁰C, a third when compared to the same situation using ECM. Although the difference between LSTM-VM and GRU-VM is not that steep.

No description available.

In this dissertation, we are interested in studying processes which generate events repeatedly over the follow-up time of a given subject. Such processes are called recurrent event processes and the data they provide are referred to as recurrent event data. Examples include the cancer recurrences, recurrent infections or disease episodes, hospital readmissions, the filing of warranty claims, and insurance claims for policy holders. In particular, we focus on the multi-type recurrent event times which usually arise when two or more different kinds of events may occur repeatedly over a period of observation. Our main objectives are to describe features of each marginal process simultaneously and study the dependence among different types of events. We present applications to a real dataset collected from the Nutritional Prevention of Cancer Trial. The objective of the clinical trial was to evaluate the efficacy of Selenium in preventing the recurrence of several types of skin cancer among 1312 residents of the Eastern United States. Four chapters are involved in this dissertation. Chapter 1 introduces a brief background to the statistical techniques used to develop the proposed methodology. We cover some concepts and useful functions related to survival data analysis and present a short introduction to frailty distributions. The Monte Carlo expectation maximization (MCEM) algorithm and copula functions for the multivariate variables are also presented in this chapter. Chapter 2 develops a multi-type recurrent events model with multivariate Gaussian random effects (frailties) for the intensity functions. In this chapter, we present nonparametric baseline intensity functions and a multivariate Gaussian distribution for the multivariate correlated random effects. An MCEM algorithm with MCMC routines in the E-step is adopted for the partial likelihood to estimate model parameters. Equations for the variances of the estimates are derived and variances of estimates are computed by Louis' formula. Predictions of the individual random effects are obtained because in some applications the magnitude of the random effects is of interest for a better understanding and interpretation of the variability in the data. The performance of the proposed methodology is evaluated by simulation studies, and the developed model is applied to the skin cancer dataset. Chapter 3 presents copula-based semiparametric multivariate frailty models for multi-type recurrent event data with applications to the skin cancer data. In this chapter, we generalize the multivariate Gaussian assumption of the frailty terms and allow the frailty distributions to have more features than the symmetric, unimodal properties of the Gaussian density. More flexible approaches to modeling the correlated frailty, referred to as copula functions, are introduced. Copula functions provide tremendous flexibility especially in allowing taking the advantages of a variety of choices for the marginal distributions and correlation structures. Semiparametric intensity models for multi-type recurrent events based on a combination of the MCEM with MCMC sampling methods and copula functions are introduced. The combination of the MCEM approach and copula function is flexible and is a generally applicable approach for obtaining inferences of the unknown parameters for high dimension frailty models. Estimation procedures for fixed effects, nonparametric baseline intensity functions, copula parameters, and predictions for the subject-specific multivariate frailties and random effects are obtained. Louis' formula for variance estimates are derived and calculated. We investigate the impact of the specification of the frailty and random effect models on the inference of covariate effects, cumulative baseline intensity functions, prediction of random effects and frailties, and the estimation of the variance-covariance components. Performances of proposed models are evaluated by simulation studies. Applications are illustrated through the dataset collected from the clinical trial of patients with skin cancer. Conclusions and some remarks for future work are presented in Chapter 4. / Ph. D.

Multiple biomarkers of oxidative stress have been measured and used in human medicine to diagnose and monitor airway disease. The purpose of the study was to determine if similar relationships existed between inflammatory and oxidative stress biomarkers in exhaled breath condensate (EBC), bronchoalveolar lavage fluid (BALF), red blood cells, white blood cells, and plasma; and cytokine expression in airway inflammatory cells and mucosal biopsies of RAO-affected horses and their controls. Sixteen horses in pairs were used: 8 non-RAO-affected (controls) and 8 RAO-affected horses. Samples from all horses were collected at remission (S1), during environmental challenge (S2) and at recovery (S3). RAO-affected horses had significant alterations in cellular glutathione peroxidase (cGPx) activity, ascorbic acid and pH in a number of biological samples. Concentrations of 8-isoprostanes, isofurans, amino acids and mRNA expression of interleukin 4 (IL4), gamma interferon (INFγ), inducible nitric oxide synthase (iNOS), extracellular glutathione peroxidase (GPx-3), and cytosolic superoxide dismutase (SOD-1) were not significantly different or were at the limits of detection. Conductivity was measured and assessed as a potential correctional factor for respiratory fluid dilution. The alterations in biomarker concentrations demonstrate that oxidative stress is an important component of airway inflammation in RAO-affected horses. Further research is warranted in the use of biomarkers and the effects of dietary interventions. / Master of Science