The research objective of this thesis is to formulate and demonstrate a computational framework for modeling the design decisions of engineers. This framework is intended to be descriptive in nature as opposed to prescriptive or normative; the output of the model represents a plausible result of a designer's decision making process. The framework decomposes the decision into three elements: the problem statement, the designer's beliefs about the alternatives, and the designer's preferences. Multi-attribute utility theory is used to capture designer preferences for multiple objectives under uncertainty. Machine-learning techniques are used to store the designer's knowledge and to make Bayesian inferences regarding the attributes of alternatives. These models are integrated into the framework of a Markov decision process to simulate multiple sequential decisions. The overall framework enables the designer's decision problem to be transformed into an optimization problem statement; the simulated designer selects the alternative with the maximum expected utility. Although utility theory is typically viewed as a normative decision framework, the perspective in this research is that the approach can be used in a descriptive context for modeling rational and non-time critical decisions by engineering designers. This approach is intended to enable the formalisms of utility theory to be used to design human subjects experiments involving engineers in design organizations based on pairwise lotteries and other methods for preference elicitation. The results of these experiments would substantiate the selection of parameters in the model to enable it to be used to diagnose potential problems in engineering design projects. The purpose of the decision-making framework is to enable the development of a design process simulation of an organization involved in the development of a large-scale complex engineered system such as an aircraft or spacecraft. The decision model will allow researchers to determine the broader effects of individual engineering decisions on the aggregate dynamics of the design process and the resulting performance of the designed artifact itself. To illustrate the model's applicability in this context, the framework is demonstrated on three example problems: a one-dimensional decision problem, a multidimensional turbojet design problem, and a variable fidelity analysis problem. Individual utility functions are developed for designers in a requirements-driven design problem and then combined into a multi-attribute utility function. Gaussian process models are used to represent the designer's beliefs about the alternatives, and a custom covariance function is formulated to more accurately represent a designer's uncertainty in beliefs about the design attributes.

The field of Machine Learning is concerned with the development of algorithms, models and techniques that solve challenging computational problems by learning from data representative of the problem (e.g. given a set of medical images previously classified by a human expert, build a model to predict unseen images as either benign or malignant). Many important real-world problems have been formulated as supervised learning problems. The assumption is that a data set is available containing the correct output (e.g. class label or target value) for each given data point. In many application domains, obtaining the correct outputs (labels) for data points is a costly and time-consuming task. This has provided the motivation for the development of Machine Learning techniques that attempt to minimize the number of labeled data points while maintaining good generalization performance on a given problem. Active Learning is one such class of techniques and is the focus of this thesis. Active Learning algorithms select or generate unlabeled data points to be labeled and use these points for learning. If successful, an Active Learning algorithm should be able to produce learning performance (e.g test set error) comparable to an equivalent supervised learner using fewer labeled data points. Theoretical, algorithmic and experimental Active Learning research has been conducted and a number of successful applications have been demonstrated. However, the scope of many of the experimental studies on Active Learning has been relatively small and there are very few large-scale experimental evaluations of Active Learning techniques. A significant amount of performance variability exists across Active Learning experimental results in the literature. Furthermore, the implementation details and effects of experimental factors have not been closely examined in empirical Active Learning research, creating some doubt over the strength and generality of conclusions that can be drawn from such results. The Active Learning model/system used in this thesis is the Optimal Active Learning algorithm framework with Gaussian Processes for regression problems (however, most of the research questions are of general interest in many other Active Learning scenarios). Experimental and implementation details of the Active Learning system used are described in detail, using a number of regression problems and datasets of different types. It is shown that the experimental results of the system are subject to significant variability across problem datasets. The hypothesis that experimental factors can account for this variability is then investigated. The results show the impact of sampling and sizes of the datasets used when generating experimental results. Furthermore, preliminary experimental results expose performance variability across various real-world regression problems. The results suggest that these experimental factors can (to a large extent) account for the variability observed in experimental results. A novel resampling technique for Optimal Active Learning, called '3-Sets Cross-Validation', is proposed as a practical solution to reduce experimental performance variability. Further results confirm the usefulness of the technique. The thesis then proposes an extension to the Optimal Active Learning framework, to perform learning via membership queries via a novel algorithm named MQOAL. The MQOAL algorithm employs the Metropolis-Hastings Markov chain Monte Carlo (MCMC) method to sample data points for query selection. Experimental results show that MQOAL provides comparable performance to the pool-based OAL learner, using a very generic, simple MCMC technique, and is robust to experimental factors related to the MCMC implementation. The possibility of making queries in batches is also explored experimentally, with results showing that while some performance degradation does occur, it is minimal for learning in small batch sizes, which is likely to be valuable in some real-world problem domains.

The field of Machine Learning is concerned with the development of algorithms, models and techniques that solve challenging computational problems by learning from data representative of the problem (e.g. given a set of medical images previously classified by a human expert, build a model to predict unseen images as either benign or malignant). Many important real-world problems have been formulated as supervised learning problems. The assumption is that a data set is available containing the correct output (e.g. class label or target value) for each given data point. In many application domains, obtaining the correct outputs (labels) for data points is a costly and time-consuming task. This has provided the motivation for the development of Machine Learning techniques that attempt to minimize the number of labeled data points while maintaining good generalization performance on a given problem. Active Learning is one such class of techniques and is the focus of this thesis. Active Learning algorithms select or generate unlabeled data points to be labeled and use these points for learning. If successful, an Active Learning algorithm should be able to produce learning performance (e.g test set error) comparable to an equivalent supervised learner using fewer labeled data points. Theoretical, algorithmic and experimental Active Learning research has been conducted and a number of successful applications have been demonstrated. However, the scope of many of the experimental studies on Active Learning has been relatively small and there are very few large-scale experimental evaluations of Active Learning techniques. A significant amount of performance variability exists across Active Learning experimental results in the literature. Furthermore, the implementation details and effects of experimental factors have not been closely examined in empirical Active Learning research, creating some doubt over the strength and generality of conclusions that can be drawn from such results. The Active Learning model/system used in this thesis is the Optimal Active Learning algorithm framework with Gaussian Processes for regression problems (however, most of the research questions are of general interest in many other Active Learning scenarios). Experimental and implementation details of the Active Learning system used are described in detail, using a number of regression problems and datasets of different types. It is shown that the experimental results of the system are subject to significant variability across problem datasets. The hypothesis that experimental factors can account for this variability is then investigated. The results show the impact of sampling and sizes of the datasets used when generating experimental results. Furthermore, preliminary experimental results expose performance variability across various real-world regression problems. The results suggest that these experimental factors can (to a large extent) account for the variability observed in experimental results. A novel resampling technique for Optimal Active Learning, called '3-Sets Cross-Validation', is proposed as a practical solution to reduce experimental performance variability. Further results confirm the usefulness of the technique. The thesis then proposes an extension to the Optimal Active Learning framework, to perform learning via membership queries via a novel algorithm named MQOAL. The MQOAL algorithm employs the Metropolis-Hastings Markov chain Monte Carlo (MCMC) method to sample data points for query selection. Experimental results show that MQOAL provides comparable performance to the pool-based OAL learner, using a very generic, simple MCMC technique, and is robust to experimental factors related to the MCMC implementation. The possibility of making queries in batches is also explored experimentally, with results showing that while some performance degradation does occur, it is minimal for learning in small batch sizes, which is likely to be valuable in some real-world problem domains.

MATTOS, C. L. C. Recurrent gaussian processes and robust dynamical modeling. 2017. 189 f. Tese (Doutorado em Engenharia de Teleinformática)–Centro de Tecnologia, Universidade Federal do Ceará, Fortaleza, 2017. / Submitted by Renato Vasconcelos (ppgeti@ufc.br) on 2017-09-09T02:26:38Z No. of bitstreams: 1 2017_tes_clcmattos.pdf: 5961013 bytes, checksum: fc44d8b852e28fa0e1ebe0c87389c0da (MD5) / Rejected by Marlene Sousa (mmarlene@ufc.br), reason: Prezado César; Prezado Pedro: Existe uma orientação para que normalizemos as dissertações e teses da UFC, em suas paginas pré-textuais e lista de referencias, pelas regras da ABNT. Por esse motivo, sugerimos consultar o modelo de template, para ajudá-lo nesta tarefa, disponível em: http://www.biblioteca.ufc.br/educacao-de-usuarios/templates/ Vamos agora as correções sempre de acordo com o template: 1. A partir da folha de aprovação as informações devem ser em língua inglesa. 2. A dedicatória deve ter a distancia até o final da folha observado. Veja no guia www.bibliotecas.ufc.br 3. A epígrafe deve ter a distancia até o final da folha observado. Veja no guia www.bibliotecas.ufc.br 4. As palavras List of Figures, LIST OF ALGORITHMS, List of Tables, Não devem ter caixa delimitando e nem ser na cor vermelha. 5. O sumário Não deve ter caixa delimitando e nem ser na cor vermelha. Nas seções terciárias, os dígitos também ficam em itálico. Os Apêndices e seus títulos, devem ficar na mesma margem da Palavra Referencias e devem iniciar com APENDICE A - Seguido do titulo. Após essas correções, enviaremos o nada consta por e-mail. Att. Marlene Rocha mmarlene@ufc.br on 2017-09-11T13:44:25Z (GMT) / Submitted by Renato Vasconcelos (ppgeti@ufc.br) on 2017-09-11T20:04:00Z No. of bitstreams: 1 2017_tes_clcmattos.pdf: 6102703 bytes, checksum: 34d9e125c70f66ca9c095e1bc6bfb7e7 (MD5) / Rejected by Marlene Sousa (mmarlene@ufc.br), reason: Lincoln, Falta apenas vc colocar no texto em português a palavra RESUMO (nesse caso não é traduzido pois se refere ao resumo em língua portuguesa) pois vc colocou ABSTRACT duas vezes para o texto em português e inglês. on 2017-09-12T11:06:29Z (GMT) / Submitted by Renato Vasconcelos (ppgeti@ufc.br) on 2017-09-12T14:05:11Z No. of bitstreams: 1 2017_tes_clcmattos.pdf: 6102699 bytes, checksum: 0a85b8841d77f0685b1153ee8ede0d85 (MD5) / Approved for entry into archive by Marlene Sousa (mmarlene@ufc.br) on 2017-09-12T16:29:17Z (GMT) No. of bitstreams: 1 2017_tes_clcmattos.pdf: 6102699 bytes, checksum: 0a85b8841d77f0685b1153ee8ede0d85 (MD5) / Made available in DSpace on 2017-09-12T16:29:18Z (GMT). No. of bitstreams: 1 2017_tes_clcmattos.pdf: 6102699 bytes, checksum: 0a85b8841d77f0685b1153ee8ede0d85 (MD5) Previous issue date: 2017-08-25 / The study of dynamical systems is widespread across several areas of knowledge. Sequential data is generated constantly by different phenomena, most of them we cannot explain by equations derived from known physical laws and structures. In such context, this thesis aims to tackle the task of nonlinear system identification, which builds models directly from sequential measurements. More specifically, we approach challenging scenarios, such as learning temporal relations from noisy data, data containing discrepant values (outliers) and large datasets. In the interface between statistics, computer science, data analysis and engineering lies the machine learning community, which brings powerful tools to find patterns from data and make predictions. In that sense, we follow methods based on Gaussian Processes (GP), a principled, practical, probabilistic approach to learning in kernel machines. We aim to exploit recent advances in general GP modeling to bring new contributions to the dynamical modeling exercise. Thus, we propose the novel family of Recurrent Gaussian Processes (RGPs) models and extend their concept to handle outlier-robust requirements and scalable stochastic learning. The hierarchical latent (non-observed) structure of those models impose intractabilities in the form of non-analytical expressions, which are handled with the derivation of new variational algorithms to perform approximate deterministic inference as an optimization problem. The presented solutions enable uncertainty propagation on both training and testing, with focus on free simulation. We comprehensively evaluate the proposed methods with both artificial and real system identification benchmarks, as well as other related dynamical settings. The obtained results indicate that the proposed approaches are competitive when compared to the state of the art in the aforementioned complicated setups and that GP-based dynamical modeling is a promising area of research. / O estudo dos sistemas dinâmicos encontra-se disseminado em várias áreas do conhecimento. Dados sequenciais são gerados constantemente por diversos fenômenos, a maioria deles não passíveis de serem explicados por equações derivadas de leis físicas e estruturas conhecidas. Nesse contexto, esta tese tem como objetivo abordar a tarefa de identificação de sistemas não lineares, por meio da qual são obtidos modelos diretamente a partir de observações sequenciais. Mais especificamente, nós abordamos cenários desafiadores, tais como o aprendizado de relações temporais a partir de dados ruidosos, dados contendo valores discrepantes (outliers) e grandes conjuntos de dados. Na interface entre estatísticas, ciência da computação, análise de dados e engenharia encontra-se a comunidade de aprendizagem de máquina, que fornece ferramentas poderosas para encontrar padrões a partir de dados e fazer previsões. Nesse sentido, seguimos métodos baseados em Processos Gaussianos (PGs), uma abordagem probabilística prática para a aprendizagem de máquinas de kernel. A partir de avanços recentes em modelagem geral baseada em PGs, introduzimos novas contribuições para o exercício de modelagem dinâmica. Desse modo, propomos a nova família de modelos de Processos Gaussianos Recorrentes (RGPs, da sigla em inglês) e estendemos seu conceito para lidar com requisitos de robustez a outliers e aprendizagem estocástica escalável. A estrutura hierárquica e latente (não-observada) desses modelos impõe expressões não- analíticas, que são resolvidas com a derivação de novos algoritmos variacionais para realizar inferência determinista aproximada como um problema de otimização. As soluções apresentadas permitem a propagação da incerteza tanto no treinamento quanto no teste, com foco em realizar simulação livre. Nós avaliamos em detalhe os métodos propostos com benchmarks artificiais e reais da área de identificação de sistemas, assim como outras tarefas envolvendo dados dinâmicos. Os resultados obtidos indicam que nossas propostas são competitivas quando comparadas ao estado da arte, mesmo nos cenários que apresentam as complicações supracitadas, e que a modelagem dinâmica baseada em PGs é uma área de pesquisa promissora.

Orientador: Ronaldo Dias / Dissertação (mestrado) - Universidade Estadual de Campinas, Instituto de Matematica, Estatistica e Computação Cientifica / Made available in DSpace on 2018-08-13T14:28:48Z (GMT). No. of bitstreams: 1 Gomes_JoseCleltoBarros_M.pdf: 1798618 bytes, checksum: db671b29b83f0321e8dbc03c5af42cde (MD5) Previous issue date: 2009 / Resumo: O desafio na modelagem de processos espaciais está na descrição da estrutura de covariância do fenômeno sob estudo. Um estimador não-paramétrico da função de covariância foi construído de forma a usar combinações lineares de funções B-splines. Estas bases são usadas com muita frequência na literatura graças ao seu suporte compacto e a computação tão rápida quanto a habilidade de criar aproximações suaves e apropriadas. Verificouse que a função de covariância estimada era definida positiva por meio do teorema de Bochner. Para a estimação da função de covariância foi implementado um algoritmo que fornece um procedimento completamente automático baseado no número de funções bases. Então foram realizados estudos numéricos que evidenciaram que assintoticamente o procedimento é consistente, enquanto que para pequenas amostras deve-se considerar as restrições das funções de covariância. As funções de covariâncias usadas na estimação foram as de exponencial potência, gaussiana, cúbica, esférica, quadrática racional, ondular e família de Matérn. Foram estimadas ainda covariâncias encaixadas. Simulações foram realizadas também a fim de verificar o comportamento da distribuição da afinidade. As estimativas apresentaram-se satisfatórias / Abstract: The challenge in modeling of spatials processes is in description of the framework of covariance of the phenomenon about study. The estimation of covariance functions was done using a nonparametric linear combinations of basis functions B-splines. These bases are used frequently in literature thanks to its compact support and fast computing as the ability to create smooth and appropriate approaches There was positive definiteness of the estimator proposed by the Bochner's theorem. For the estimation of the covariance functions was implemented an algorithm that provides a fully automated procedure based on the number of basis functions. Then numerical studies were performed that showed that the procedure is consistent assynthotically. While for small samples should consider the restrictions of the covariance functions, so the process of optimization was non-linear optimization with restrictions. The following covariance functions were used in estimating: powered exponential, Gaussian, cubic, spherical, rational quadratic and Matérn family. Nested covariance funtions still were estimated. Simulations were also performed to verify the behavior of affinity and affinity partial, which measures how good is the true function of the estimated function. Estimates showed satisfactory / Mestrado / Mestre em Estatística

In recent years, the level of technology in heavy duty vehicles has increased significantly. Progress has been made towards autonomous driving, with increaseddriver comfort and safety, partly by use of advanced driver assistance systems (ADAS). In this thesis the possibilities to detect and predict lane changes for the preceding vehicle are studied. This important information will help to improve the decision-making for safety systems. Some suitable approaches to solving the problem are presented, along with an evaluation of their related accuracies. The modelling of human perceptions and actions is a challenging task. Several thousand kilometers of driving data was available, and a reasonable course of action was to let the system learn from this off-line. For the thesis it was therefore decided to review the possibility to utilize a branch within the area of artificial intelligence, called supervised learning. The study of driving intentions was formulatedas a binary classification problem. To distinguish between lane-change and lane-keep actions, four machine learning-techniques were evaluated, namely naive Bayes, artificial neural networks, support vector machines and Gaussian processes. As input to the classifiers, fused sensor signals from today commercially accessible systems in Scania vehicles were used. The project was carried out within the boundaries of a Master’s Thesis projectin collaboration between Linköping University and Scania CV AB. Scania CV AB is a leading manufacturer of heavy trucks, buses and coaches, alongside industrialand marine engines.

This thesis presents methods for the inference of system state and the learning of model structure for a number of hidden-state time series models, within a Bayesian probabilistic framework. Motivating examples are taken from application areas including finance, physical object tracking and audio restoration. The work in this thesis can be broadly divided into three themes: system and parameter estimation in linear jump-diffusion systems, non-parametric model (system) estimation and batch audio restoration. For linear jump-diffusion systems, efficient state estimation methods based on the variable rate particle filter are presented for the general linear case (chapter 3) and a new method of parameter estimation based on Particle MCMC methods is introduced and tested against an alternative method using reversible-jump MCMC (chapter 4). Non-parametric model estimation is examined in two settings: the estimation of non-parametric environment models in a SLAM-style problem, and the estimation of the network structure and forms of linkage between multiple objects. In the former case, a non-parametric Gaussian process prior model is used to learn a potential field model of the environment in which a target moves. Efficient solution methods based on Rao-Blackwellized particle filters are given (chapter 5). In the latter case, a new way of learning non-linear inter-object relationships in multi-object systems is developed, allowing complicated inter-object dynamics to be learnt and causality between objects to be inferred. Again based on Gaussian process prior assumptions, the method allows the identification of a wide range of relationships between objects with minimal assumptions and admits efficient solution, albeit in batch form at present (chapter 6). Finally, the thesis presents some new results in the restoration of audio signals, in particular the removal of impulse noise (pops and clicks) from audio recordings (chapter 7).

Visual speech synthesis deals with synthesising facial animation from an audio representation of speech. In the last decade or so, data-driven approaches have gained prominence with the development of Machine Learning techniques that can learn an audio-visual mapping. Many of these Machine Learning approaches learn a generative model of speech production using the framework of probabilistic graphical models, through which efficient inference algorithms can be developed for synthesis. In this work, the audio and visual parameters are assumed to be generated from an underlying latent space that captures the shared information between the two modalities. These latent points evolve through time according to a dynamical mapping and there are mappings from the latent points to the audio and visual spaces respectively. The mappings are modelled using Gaussian processes, which are non-parametric models that can represent a distribution over non-linear functions. The result is a non-linear state-space model. It turns out that the state-space model is not a very accurate generative model of speech production because it assumes a single dynamical model, whereas it is well known that speech involves multiple dynamics (for e.g. different syllables) that are generally non-linear. In order to cater for this, the state-space model can be augmented with switching states to represent the multiple dynamics, thus giving a switching state-space model. A key problem is how to infer the switching states so as to model the multiple non-linear dynamics of speech, which we address by learning a variable-order Markov model on a discrete representation of audio speech. Various synthesis methods for predicting visual from audio speech are proposed for both the state-space and switching state-space models. Quantitative evaluation, involving the use of error and correlation metrics between ground truth and synthetic features, is used to evaluate our proposed method in comparison to other probabilistic models previously applied to the problem. Furthermore, qualitative evaluation with human participants has been conducted to evaluate the realism, perceptual characteristics and intelligibility of the synthesised animations. The results are encouraging and demonstrate that by having a joint probabilistic model of audio and visual speech that caters for the non-linearities in audio-visual mapping, realistic visual speech can be synthesised from audio speech.

<div>Physical phenomena in nature are typically represented by complex systems of ordinary differential equations (ODEs) or partial differential equations (PDEs), modeling a wide range of spatio-temporal scales and multi-physics. The field of computational science has achieved indisputable success in advancing our understanding of the natural world - made possible through a combination of increasingly sophisticated mathematical models, numerical techniques and hardware resources. Furthermore, there has been a recent revolution in the data-driven sciences - spurred on by advances in the deep learning/stochastic optimization communities and the democratization of machine learning (ML) software.</div><div><br></div><div><div>With the ubiquity of use of computational models for analysis and prediction of physical systems, there has arisen a need for rigorously characterizing the effects of unknown variables in a system. Unfortunately, Uncertainty quantification (UQ) tasks such as model calibration, uncertainty propagation, and optimization under uncertainty, typically require several thousand evaluations of the underlying physical models. In order to deal with the high cost of the forward model, one typically resorts to the surrogate idea - replacing the true response surface with an approximation that is both accurate as well cheap (computationally speaking). However, state-ofart numerical systems are often characterized by a very large number of stochastic parameters - of the order of hundreds or thousands. The high cost of individual evaluations of the forward model, coupled with the limited real world computational budget one is constrained to work with, means that one is faced with the task of constructing a surrogate model for a system with high input dimensionality and small dataset sizes. In other words, one faces the <i>curse of dimensionality</i>.</div></div><div><br></div><div><div>In this dissertation, we propose multiple ways of overcoming the<i> curse of dimensionality</i> when constructing surrogate models for high-dimensional numerical simulators. The core idea binding all of our proposed approach is simple - we try to discover special structure in the stochastic parameter which captures most of the variance of the output quantity of interest. Our strategies first identify such a low-rank structure, project the high-dimensional input onto it, and then link the projection to the output. If the dimensionality of the low dimensional structure is small enough, learning the map between this reduced input space to the output is a much easier task in</div><div>comparison to the original surrogate modeling task.</div></div>

This thesis is concerned with model predictive control (MPC) within the field of autonomous driving. MPC requires a model of the system to be controlled. Since a vehicle is expected to handle a wide range of driving conditions, it is crucial that the model of the vehicle dynamics is able to account for this. Differences in road grip caused by snowy, icy or muddy roads change the driving dynamics and relying on a single model, based on ideal conditions, could possibly lead to dangerous behaviour. This work investigates the use of Gaussian processes for learning a model that can account for varying road friction coefficients. This model is incorporated as an extension to a nominal vehicle model. A double lane change scenario is considered and the aim is to learn a GP model of the disturbance based on previous driving experiences with a road friction coefficient of 0.4 and 0.6 performed with a regular MPC controller. The data is then used to train a GP model. The GPMPC controller is then compared with the regular MPC controller in the case of trajectory tracking. The results show that the obtained GP models in most cases correctly predict the model error in one prediction step. For multi-step predictions, the results vary more with some cases showing an improved prediction with a GP model compared to the nominal model. In all cases, the GPMPC controller gives a better trajectory tracking than the MPC controller while using less control input.