This thesis applies the methods of previous work in aerial magnetic field mapping and use in state estimation to the Virginia Tech Swing Space motion capture indoor facility. State estimation with magnetic field data acquired from a quadrotor is comparatively performed with Gaussian process regression, a multiplicative extended Kalman filter, and a particle filter to estimate the position and attitude of an uncrewed aircraft system (UAS) at any point in the motion capture testing environment. Motion capture truth data is used in the analysis. The first experimental method utilized in this thesis is Gaussian process regression. This machine learning tool allows us to create three-dimensional magnetic field maps of the indoor test space by collecting magnetic field vector data with a small UAS. Here, the maps illustrate the 3D magnetic field strengths and directions in the Virginia Tech Swing Space motion capture lab. Also, the magnetic field spatial variation of the test space is analyzed, yielding higher magnetic field gradient at lower heights above the ground. Next, the multiplicative extended Kalman filter is used with our Gaussian process regression magnetic field maps to estimate the attitude of the quadrotor. The results indicate an increase in attitude estimation accuracy when magnetic field mapping is utilized compared to when it is not. Here, results show that the addition of aerial magnetic field mapping leads to enhanced attitude estimation. Finally, the particle filter is utilized with support from our magnetic field maps to estimate the position of a small quadrotor UAS. The magnetic field maps allow us to obtain UAS position vectors by tracking UAS movement through magnetic field data. The particle filter gives three-dimensional position estimates to within 0.2 meters for five out of our eight test flights. The root mean square error is within 0.1 meters for each test flight. The effects of magnetic field spatial variation are also analyzed. The accuracy of position estimation is higher for two out the four flights in the maximum magnetic gradient area, while the accuracy is similar in both minimum and maximum gradient regions for the remaining two flights. There is evidence to support an increase in accuracy for high magnetic variation areas, but further work is needed to confirm utility for practical applications. / Master of Science / This thesis investigates airborne magnetic field mapping for the Virginia Tech Swing Space motion capture indoor facility. Position and attitude estimation with magnetic field data acquired from a small uncrewed aircraft system (UAS) is comparatively performed with multiple estimation methods. Motion capture truth data is used in analyses. The first data processing method is called Gaussian process regression. This machine learning tool allows us to create magnetic field maps of the indoor test space by averaging or regressing field estimates over collected UAS data. The maps illustrate the magnetic field strengths and directions over a three dimensional volume in the Virginia Tech Swing Space motion capture lab. Next, a multiplicative extended Kalman filter is used with our Gaussian process regression magnetic field maps to estimate UAS attitude. Results show improvement in attitude estimation accuracy when magnetic field mapping is utilized compared to when it is not. Finally, a particle filter method is utilized with our magnetic field maps to estimate UAS position. The particle filter estimates three-dimensional UAS position estimates to within 0.2 meters for five out of our eight test flights. The effects of magnetic field spatial variation are also analyzed, indicating the need for future work before magnetic field based position estimation can be practically applied.

The proliferation of renewables, electric vehicles, and power electronic devices calls for innovative approaches to learn, optimize, and plan the power system. The uncertain and volatile nature of the integrated components necessitates using swift and probabilistic solutions. Gaussian process regression is a machine learning paradigm that provides closed-form predictions with quantified uncertainties. The key property of Gaussian processes is the natural ability to integrate the sensitivity of the labels with respect to features, yielding improved accuracy. This dissertation tailors Gaussian process regression for three applications in power systems. First, a physics-informed approach is introduced to infer the grid dynamics using synchrophasor data with minimal network information. The suggested method is useful for a wide range of applications, including prediction, extrapolation, and anomaly detection. Further, the proposed framework accommodates heterogeneous noisy measurements with missing entries. Second, a learn-to-optimize scheme is presented using Gaussian process regression that predicts the optimal power flow minimizers given grid conditions. The main contribution is leveraging sensitivities to expedite learning and achieve data efficiency without compromising computational efficiency. Third, Bayesian optimization is applied to solve a bi-level minimization used for strategic investment in electricity markets. This method relies on modeling the cost of the outer problem as a Gaussian process and is applicable to non-convex and hard-to-evaluate objective functions. The designed algorithm shows significant improvement in speed while attaining a lower cost than existing methods. / Doctor of Philosophy / The proliferation of renewables, electric vehicles, and power electronic devices calls for innovative approaches to learn, optimize, and plan the power system. The uncertain and volatile nature of the integrated components necessitates using swift and probabilistic solutions. This dissertation focuses on three practically important problems stemming from the power system modernization. First, a novel approach is proposed that improves power system monitoring, which is the first and necessary step for the stable operation of the network. The suggested method applies to a wide range of applications and is adaptable to use heterogeneous and noisy measurements with missing entries. The second problem focuses on predicting the minimizers of an optimization task. Moreover, a computationally efficient framework is put forth to expedite this process. The third part of this dissertation identifies investment portfolios for electricity markets that yield maximum revenue and minimum cost.

Gaussian process regression (GPR) is a long-standing technique for statistical interpolation between observed data points. Having originally been applied to spatial analysis in the 1950s, GPR offers highly nonlinear predictions with uncertainty adjusting to the degree of extrapolation -- at the expense of very few model parameters to be fit. Thus GPR has gained considerable popularity in statistical applications such as machine learning and nonparametric density estimation. In this thesis, we explore the potential for GPR to improve the efficiency of parametric inference for continuous-time stochastic processes. For almost all such processes, the likelihood function based on discrete observations cannot be written in closed-form. However, it can be very well approximated if the inter-observation time is small. Therefore, a popular strategy for parametric inference is to introduce missing data between actual observations. In a Bayesian context, samples from the posterior distribution of the parameters and missing data are then typically obtained using Markov chain Monte Carlo (MCMC) methods, which can be computationally very expensive. Here, we consider the possibility of using GPR to impute the marginal distribution of the missing data directly. These imputations could then be leveraged to produce independent draws from the joint posterior by Importance Sampling, for a significant gain in computational efficiency. In order to illustrate the methodology, three continuous processes are examined. The first one is based on a neural excitation model with a non-standard periodic component. The second and third are popular financial models often used for option pricing. While preliminary inferential results are quite promising, we point out several improvements to the methodology which remain to be explored.

No description available.

Increases in demand have driven the development of complex processing technology for separating mineral resources from exceedingly low grade multi- component resources. Low mineral concentrations and variable feedstocks can make separating signal from noise difficult, while high process complexity and the multi-component nature of a feedstock can make testwork, optimization, and process simulation difficult or infeasible. A prime example of such a scenario is the recovery and separation of rare earth elements (REEs) and other critical minerals from acid mine drainage (AMD) using a solvent extraction (SX) process. In this process the REE concentration found in an AMD source can vary site to site, and season to season. SX processes take a non-trivial amount of time to reach steady state. The separation of numerous individual elements from gangue metals is a high-dimensional problem, and SX simulators can have a prohibitive computation time. Bayesian statistical methods intrinsically quantify uncertainty of model parameters and predictions given a set of data and a prior distribution and model parameter prior distributions. The uncertainty quantification possible with Bayesian methods lend well to statistical simulation, model selection, and sensitivity analysis. Moreover, Bayesian models utilizing Gaussian Process priors can be used for active learning tasks which allow for prediction, optimization, and simulator calibration while reducing data requirements. However, literature on Bayesian methods applied to separations engineering is sparse. The goal of this dissertation is to investigate, illustrate, and test the use of a handful of Bayesian methods applied to process engineering problems. First further details for the background and motivation are provided in the introduction. The literature review provides further information regarding critical minerals, solvent extraction, Bayeisan inference, data reconciliation for separations, and Gaussian process modeling. The body of work contains four chapters containing a mixture of novel applications for Bayesian methods and a novel statistical method derived for the use with the motivating problem. Chapter topics include Bayesian data reconciliation for processes, Bayesian inference for a model intended to aid engineers in deciding if a process has reached steady state, Bayesian optimization of a process with unknown dynamics, and a novel active learning criteria for reducing the computation time required for the Bayesian calibration of simulations to real data. In closing, the utility of a handfull of Bayesian methods are displayed. However, the work presented is not intended to be complete and suggestions for further improvements to the application of Bayesian methods to separations are provided. / Doctor of Philosophy / Rare earth elements (REEs) are a set of elements used in the manufacture of supplies used in green technologies and defense. Demand for REEs has prompted the development of technology for recovering REEs from unconventional resources. One unconventional resource for REEs under investigation is acid mine drainage (AMD) produced from the exposure of certain geologic strata as part of coal mining. REE concentrations found in AMD are significant, although low compared to REE ore, and can vary from site to site and season to season. Solvent extraction (SX) processes are commonly utilized to concentrate and separate REEs from contaminants using the differing solubilities of specific elements in water and oil based liquid solutions. The complexity and variability in the processes used to concentrate REEs from AMD with SX motivates the use of modern statistical and machine learning based approaches for filtering noise, uncertainty quantification, and design of experiments for testwork, in order to find the truth and make accurate process performance comparisons. Bayesian statistical methods intrinsically quantify uncertainty. Bayesian methods can be used to quantify uncertainty for predictions as well as select which model better explains a data set. The uncertainty quantification available with Bayesian models can be used for decision making. As a particular example, the uncertainty quantification provided by Gaussian process regression lends well to finding what experiments to conduct, given an already obtained data set, to improve prediction accuracy or to find an optimum. However, literature is sparse for Bayesian statistical methods applied to separation processes. The goal of this dissertation is to investigate, illustrate, and test the use of a handful of Bayesian methods applied to process engineering problems. First further details for the background and motivation are provided in the introduction. The literature review provides further information regarding critical minerals, solvent extraction, Bayeisan inference, data reconciliation for separations, and Gaussian process modeling. The body of work contains four chapters containing a mixture of novel applications for Bayesian methods and a novel statistical method derived for the use with the motivating problem. Chapter topics include Bayesian data reconciliation for processes, Bayesian inference for a model intended to aid engineers in deciding if a process has reached steady state, Bayesian optimization of a process with unknown dynamics, and a novel active learning criteria for reducing the computation time required for the Bayesian calibration of simulations to real data. In closing, the utility of a handfull of Bayesian methods are displayed. However, the work presented is not intended to be complete and suggestions for further improvements to the application of Bayesian methods to separations are provided.

The establishment of the General Data Protection Regulation (GDPR) in Europe in 2018, including the "Right to be Forgotten" poses important questions about the necessity of efficient data deletion techniques for trained Machine Learning models to completely enforce this right, since retraining from scratch such models whenever a data point must be deleted seems impractical. We tackle such a problem for Gaussian Process Regression and define in this paper an efficient exact unlearning technique for Gaussian Process Regression which completely include the optimization of the hyperparameters of the kernel function. The method is based on an efficient retracing of past optimizations by the Resilient Backpropagation (Rprop) algorithm through the online formulation of a Gaussian Process regression. Furthermore, we develop an extension of the proposed method to the Product-of-Experts and Bayesian Committee Machines types of local approximations of Gaussian Process Regression, further enhancing the unlearning capabilities through a random partitioning of the dataset. The performance of the proposed method is largely dependent on the regression task. We show through multiple experiments on different problems that several iterations of such optimization can be recomputed without any need for kernel matrix inversions, at the cost of saving intermediate states of the training phase. We also offer different ideas to extend this method to an approximate unlearning scheme, even further improving its computational complexity. / Införandet av Dataskyddsförordningen (DSF) i Europa 2018, inklusive rätten att bli bortglömd, ställer viktiga frågor om nödvändigheten av effektiva dataraderingtekniker för tränade maskininlärningsmodeller för att följa denna rättighet, detta eftersom omskolning från grunden av tränade modeller när en datapunkt måste raderas verkar opraktiskt. Vi tacklar dataraderingsproblemet för regression av Gaussiska processer och vi definierar i detta dokument en effektiv exakt avlärningsteknik för Gaussisk process regression som inkluderar optimeringen av kärnfunktionens hyperparametrarna. Metoden är baserad på en effektiv omberäkning av tidigare optimeringar genom Resilient Backpropagation (Rprop)-algoritmen tack vare onlineformuleringen medelst en Gaussisk processregression. Dessutom utvecklar vi en utvidgning av den föreslagna metoden till produkter-av-experter och Bayesianska kommittémaskiner av lokala approximationer av Gaussiska processregression, för att förbättra avlärningskapaciteten genom att använda en slumpmässig partitionering av datasetet. Metodernas prestanda beror till stor del på regressionsuppgiften. Vi visar med flera experiment på olika problem att flera iterationer av optimeringarna kan omberäknas utan behov av kärnmatrisinversioner, men på bekostnad av att spara mellanstatus i träningsfasen. Vi föreslår också olika idéer för att utvidga denna metod till en approximativ avlärningsteknik, för att förbättra dess beräkningskomplexitet. / L’établissement du Règlement Général sur la Protection des Données (RGPD) en Europe en 2018, incluant le "Droit à l’Oubli" pose de sérieuses questions vis-à-vis de l’importance du développement de techniques permettant le "désapprentissage" de données specifiques d’un modéle entrainé. Réentrainer un modèle "from scratch" dés qu’une donnée doit être supprimée pose problème en pratique, ce qui justifie le besoin de méthodes plus efficaces pour répondre à ce problème. Nous abordons ce problème dans le contexte d’une Gaussian Process Regression, et définissons dans ce rapport une méthode efficace et exacte de désapprentissage pour une Gaussian Process Regression incluant l’optimisation des hyperparamètres du noyau. La méthode est basée sur un traçage efficace de l’optimisation faite par l’algorithme de Resilient Backpropagation (Rprop) grâce à la formulation Online d’une Gaussian Process Regression. De plus, nous développons une extension de cette première méthode pour la rendre applicable à des approximations locales telles que les Product-of-Experts ou Bayesian Committee Machines, ce qui permet d’améliorer d’avantage les performance de désapprentissage grâce à partitionement aléatoire du jeu de données. Du fait de la forte dépendence des performances de désapprentissage à la tâche de regression, nous montrons à travers de multiples expériences sur différents jeux de données qu’un nombre conséquent d’itérations peut être recalculé efficacement sans nécessiter d’inversion de matrices, au prix de la sauvegarde des états intermédiaires de la phase d’apprentissage.Nous donnons finalement des idées pour étendre cette méthode vers un désapprentissage approximatif, afin d’améliorer une fois de plus le temps de désapprentissage.

Tato práce se zabývá úlohou odhadování prostorové funkce z hlediska regrese pomocí Gaussovských procesů (GPR) za současné nejistoty tréninkových pozic (pozic senzorů). Nejdříve je zde popsána teorie v pozadí GPR metody pracující se známými tréninkovými pozicemi. Tato teorie je poté aplikována při odvození výrazů prediktivní distribuce GPR v testovací pozici při uvážení nejistoty tréninkových pozic. Kvůli absenci analytického řešení těchto výrazů byly výrazy aproximovány pomocí metody Monte Carlo. U odvozené metody bylo demonstrováno zlepšení kvality odhadu prostorové funkce oproti standardnímu použití GPR metody a také oproti zjednodušenému řešení uvedenému v literatuře. Dále se práce zabývá možností použití metody GPR s nejistými tréninkovými pozicemi v~kombinaci s výrazy s dostupným analytickým řešením. Ukazuje se, že k dosažení těchto výrazů je třeba zavést značné předpoklady, což má od počátku za následek nepřesnost prediktivní distribuce. Také se ukazuje, že výsledná metoda používá standardní výrazy GPR v~kombinaci s upravenou kovarianční funkcí. Simulace dokazují, že tato metoda produkuje velmi podobné odhady jako základní GPR metoda uvažující známé tréninkové pozice. Na druhou stranu prediktivní variance (nejistota odhadu) je u této metody zvýšena, což je žádaný efekt uvážení nejistoty tréninkových pozic.

The purpose of this thesis is to investigate how data from a residential property owner can be utilized to enable better energy management for their building stock. Specifically, this is done through the development of two machine learning models with the objective of detecting anomalies in the existing data of electricity consumption. The dataset consists of two years of residential electricity consumption for 193 substations belonging to the residential property owner Uppsalahem. The first of the developed models uses the K-means method to cluster substations with similar consumption patterns to create electricity profiles, while the second model uses Gaussian process regression to predict electricity consumption of a 24 hour timeframe. The performance of these models is evaluated and the optimal models resulting from this process are implemented to detect anomalies in the electricity consumption data. Two different algorithms for anomaly detection are presented, based on the differing properties of the two earlier models. During the evaluation of the models, it is established that the consumption patterns of the substations display a high variability, making it difficult to accurately model the full dataset. Both models are shown to be able to detect anomalies in the electricity consumption data, but the K-means based anomaly detection model is preferred due to it being faster and more reliable. It is concluded that substation electricity consumption is not ideal for anomaly detection, and that if a model should be implemented, it should likely exclude some of the substations with less regular consumption profiles.

Metamodeling is regarded as a powerful analysis tool to learn the input-output relationship of a system based on a limited amount of data collected when experiments with real systems are costly or impractical. As a popular metamodeling method, Gaussian process regression (GPR), has been successfully applied to analyses of various engineering systems. However, GPR-based metamodeling for time-dependent systems (TDSs) is especially challenging due to three reasons. First, TDSs require an appropriate account for temporal effects, however, standard GPR cannot address temporal effects easily and satisfactorily. Second, TDSs typically require analytics tools with a sufficiently high computational efficiency to support online decision making, but standard GPR may not be adequate for real-time implementation. Lastly, reliable uncertainty quantification is a key to success for operational planning of TDSs in real world, however, research on how to construct adequate error bounds for GPR-based metamodeling is sparse. Inspired by the challenges encountered in GPR-based analyses of two representative stochastic TDSs, i.e., load forecasting in a power system and trajectory prediction for unmanned aerial vehicles (UAVs), this dissertation aims to develop novel modeling, sampling, and statistical analysis techniques for enhancing the computational and statistical efficiencies of GPR-based metamodeling to meet the requirements of practical implementations. Furthermore, an in-depth investigation on building uniform error bounds for stochastic kriging is conducted, which sets up a foundation for developing robust GPR-based metamodeling techniques for analyses of TDSs under the impact of strong heteroscedasticity. / Ph.D. / Metamodeling has been regarded as a powerful analysis tool to learn the input-output relationship of an engineering system with a limited amount of experimental data available. As a popular metamodeling method, Gaussian process regression (GPR) has been widely applied to analyses of various engineering systems whose input-output relationships do not depend on time. However, GPR-based metamodeling for time-dependent systems (TDSs), whose input-output relationships depend on time, is especially challenging due to three reasons. First, standard GPR cannot properly address temporal effects for TDSs. Second, standard GPR is typically not computationally efficient enough for real-time implementations in TDSs. Lastly, research on how to adequately quantify the uncertainty associated with the performance of GPR-based metamodeling is sparse. To fill this knowledge gap, this dissertation aims to develop novel modeling, sampling, and statistical analysis techniques for enhancing standard GPR to meet the requirements of practical implementations for TDSs. Effective solutions are provided to address the challenges encountered in GPR-based analyses of two representative stochastic TDSs, i.e., load forecasting in a power system and trajectory prediction for unmanned aerial vehicles (UAVs). Furthermore, an in-depth investigation on quantifying the uncertainty associated with the performance of stochastic kriging (a variant of standard GPR) is conducted, which sets up a foundation for developing robust GPR-based metamodeling techniques for analyses of more complex TDSs.

京都大学 / 新制・課程博士 / 博士(情報学) / 甲第23588号 / 情博第782号 / 新制||情||133(附属図書館) / 京都大学大学院情報学研究科システム科学専攻 / (主査)教授 田中 利幸, 教授 下平 英寿, 准教授 櫻間 一徳 / 学位規則第4条第1項該当 / Doctor of Informatics / Kyoto University / DGAM