This thesis consists of four papers studying structure learning and Bayesian inference in probabilistic graphical models for both undirected and directed acyclic graphs (DAGs). Paper A presents a novel algorithm, called the Christmas tree algorithm (CTA), that incrementally construct junction trees for decomposable graphs by adding one node at a time to the underlying graph. We prove that CTA with positive probability is able to generate all junction trees of any given number of underlying nodes. Importantly for practical applications, we show that the transition probability of the CTA kernel has a computationally tractable expression. Applications of the CTA transition kernel are demonstrated in a sequential Monte Carlo (SMC) setting for counting the number of decomposable graphs. Paper B presents the SMC scheme in a more general setting specifically designed for approximating distributions over decomposable graphs. The transition kernel from CTA from Paper A is incorporated as proposal kernel. To improve the traditional SMC algorithm, a particle Gibbs sampler with a systematic refreshment step is further proposed. A simulation study is performed for approximate graph posterior inference within both log-linear and decomposable Gaussian graphical models showing efficiency of the suggested methodology in both cases. Paper C explores the particle Gibbs sampling scheme of Paper B for approximate posterior computations in the Bayesian predictive classification framework. Specifically, Bayesian model averaging (BMA) based on the posterior exploration of the class-specific model is incorporated into the predictive classifier to take full account of the model uncertainty. For each class, the dependence structure underlying the observed features is represented by a distribution over the space of decomposable graphs. Due to the intractability of explicit expression, averaging over the approximated graph posterior is performed. The proposed BMA classifier reveals superior performance compared to the ordinary Bayesian predictive classifier that does not account for the model uncertainty, as well as to a number of out-of-the-box classifiers. Paper D develops a novel prior distribution over DAGs with the ability to express prior knowledge in terms of graph layerings. In conjunction with the prior, a stochastic optimization algorithm based on the layering property of DAGs is developed for performing structure learning in Bayesian networks. A simulation study shows that the algorithm along with the prior has superior performance compared with existing priors when used for learning graph with a clearly layered structure. / <p>QC 20170915</p>

Esta dissertação flexibiliza a suposição de normalidade, dispondo de distribuições assimétricas em modelos de crescimento. Propõe uma abordagem bayesiana para ajuste de modelos não lineares quando a suposição de normalidade para os erros não é razoável e/ou apresentam heteroscedasticidade. Assim, adota-se as distribuições skew-normal e skew-t para as situações em que é necessário modelar dados com caudas mais pesadas ou mais leves que a normal e assimétricos; sendo que é considerado também a presença de heteroscedasticidade. Diferentes funções são utilizadas na estrutura multiplicativa para modelar a variância. Com esse objetivo, métodos de inferência na abordagem bayesiana são desenvolvidos para estimar os parâmetros dos modelos de regressão não linear com os erros seguindo as distribuições citadas anteriormente. A metodologia visa aplicação à curvas de crescimento para dados de árvores / This paper relaxes the assumption of normality, featuring asymmetric distributions in growth models. Proposes a Bayesian approach to fit nonlinear models when the assumption of normality for the errors is not reasonable and/or exhibit heteroscedasticity. Thus, we adopt the skew-normal and skew-t distributions for situations where it is necessary to model data with tails heavier or lighter than normal and asymmetric, which is considered also the presence of heteroscedasticity. Different functions are used to model the multiplicative structure of variance. With this objective, methods of inference in the Bayesian approach are developed to estimate the parameters of nonlinear regression models with errors following the distributions listed above. The methodology is intended to apply to the growth curves for trees data sets

Conhecer a distribuição geográfica das espécies é primordial para a tomada de ações efetivas de conservação. Modelos de ocupação são ferramentas importantes para estimar a distribuição das espécies, especialmente quando as informações são incompletas, como é o caso de muitas espécies ameaçadas ou em áreas ainda insuficientemente amostradas. O objetivo deste estudo é ampliar e refinar o conhecimento sobre a distribuição geográfica da toninha, Pontoporia blainvillei, um pequeno cetáceo ameaçado de extinção restrito às águas costeiras do Atlântico Sul ocidental, através de modelos de ocupação. Foram realizadas amostragens aéreas com 4 observadores independentes, em 2058 sítios de 4x4km na distribuição da espécie no Brasil. Foram utilizadas cinco covariáveis de detecção (transparência da água, escala Beaufort, reflexo solar, posição dos amostradores e número de amostradores) e três covariáveis de ocupação (batimetria, temperatura média e produtividade primária) com índices de correlação de Pearson menor que 0,7. Todas as covariáveis contínuas foram estandardizadas com média zero e desvio padrão igual a um. Os modelos de ocupação com autocorrealação espacial foram estimados com Inferência Bayesiana utilizando priors ‘vagos’ (média zero e variância 1.0E6). Em apenas 75 sítios foram detectadas toninhas. A probabilidade de detecção média foi de 0.23 (CRI 0.006 a 0.51), onde as covariáveis Beaufort (efeito negativo), reflexo solar (efeito negativo) e transparência da água (efeito positivo) apresentaram efeitos significativos. A média estimada de ocupação foi de 0,066 (CRI 0,01 a 0,31). As covariáveis batimetria e a temperatura média apresentaram efeitos positivos e negativos sobre o processo de ocupação, respectivamente. Espacialmente o modelo prevê três áreas com altas probabilidades de ocupação aparentemente disjuntas: a) costa norte do Rio de Janeiro; b) costas norte de 3 Santa catarina até São Paulo; c) costa do Rio Grande do Sul. Assim, agregamos importantes informações para a conservação da espécie e realização de novos estudos, apontando onde podemos encontrar maiores probabilidade de ocupação na costa do Brasil e covariáveis que determinam a ocupação e a detecção da espécie. / Knowing the geographic distribution of a species is essential for taking effective conservation actions. Occupation Models are important tools for estimating species distribution, especially when information is incomplete, as is the case with many endangered species or in under-sampled areas. The aim of this study is to expand and refine the knowledge about the geographic distribution of the franciscana, Pontoporia blainvillei, a threatened small cetacean restricted to the coastal waters of the western South Atlantic, through Occupation Models. Aerial samplings were carried out with 4 independent observers, in 2058 sites of 4x4km across the distribution of the species in Brazilian waters. Five detection covariates were used (water transparency, Beaufort scale, solar reflectance, observer position and number of observers) and three covariates of occupation (bathymetry, mean temperature and primary productivity) with Pearson correlation indices less than 0.7. All continuous covariates were standardized with mean zero and standard deviation equal to one. Occupancy Models with spatial autocorrection were estimated using Bayesian Inference using 'vague' priors (zero mean and variance 1.0E6). Franciscana was detected only in 75 sites. The average detection probability 4 was 0.23 (CRI 0.006 to 0.51), where Beaufort (negative effect), solar reflex (negative effect) and water transparency (positive effect) covariables had significant effects. The estimated mean occupancy was 0.066 (CRI 0.01 to 0.31). The bathymetry and the mean temperature covariables had positive and negative effects on the occupation process, respectively. Spatially the model predicts three apparently disjunct areas with high probability of occupation: a) north coast of Rio de Janeiro; b) north coasts of Santa Catarina to São Paulo; c) coast of Rio Grande do Sul. Thus, we add important information for the conservation of species and new studies, pointing out where we can find greater likelihood of occupation on the coast of Brazil and covariates that determine the occupation and the detection of the species.

京都大学 / 0048 / 新制・課程博士 / 博士(工学) / 甲第22653号 / 工博第4737号 / 新制||工||1740(附属図書館) / 京都大学大学院工学研究科都市社会工学専攻 / (主査)教授 山田 忠史, 教授 藤井 聡, 准教授 SCHMOECKER Jan-Dirk / 学位規則第4条第1項該当 / Doctor of Philosophy (Engineering) / Kyoto University / DFAM

Available statistical methodologies focus more on accommodating continuous variables, however recently dealing with count data has received high interest in the statistical literature. In this dissertation, we propose some statistical approaches to investigate linear and nonlinear dependencies between two discrete random variables, or between a discrete and continuous random variables. Copula functions are powerful tools for modeling dependencies between random variables. We derive copula-based population version of Spearman’s rho when at least one of the marginal distribution is discrete. In each case, the functional relationship between Kendall’s tau and Spearman’s rho is obtained. The asymptotic distributions of the proposed estimators of these association measures are derived and their corresponding confidence intervals are constructed, and tests of independence are derived. Then, we propose a Bayesian copula factor autoregressive model for time series mixed data. This model assumes conditional independence and shares latent factors in both mixed-type response and multivariate predictor variables of the time series through a quadratic timeseries regression model. This model is able to reduce the dimensionality by accommodating latent factors in both response and predictor variables of the high-dimensional time series data. A semiparametric time series extended rank likelihood technique is applied to the marginal distributions to handle mixed-type predictors of the high-dimensional time series, which decreases the number of estimated parameters and provides an efficient computational algorithm. In order to update and compute the posterior distributions of the latent factors and other parameters of the models, we propose a naive Bayesian algorithm with Metropolis-Hasting and Forward Filtering Backward Sampling methods. We evaluate the performance of the proposed models and methods through simulation studies. Finally, each proposed model is applied to a real dataset.

Embodied agents, be they animals or robots, acquire information about the world through their senses. Embodied agents, however, do not simply lose this information once it passes by, but rather process and store it for future use. The most general theory of how an agent can combine stored knowledge with new observations is Bayesian inference. In this dissertation I present a theory of how embodied agents can learn to implement Bayesian inference with neural networks. By neural network I mean both artificial and biological neural networks, and in my dissertation I address both kinds. On one hand, I develop theory for implementing Bayesian inference in deep generative models, and I show how to train multilayer perceptrons to compute approximate predictions for Bayesian filtering. On the other hand, I show that several models in computational neuroscience are special cases of the general theory that I develop in this dissertation, and I use this theory to model and explain several phenomena in neuroscience. The key contributions of this dissertation can be summarized as follows: - I develop a class of graphical model called nth-order harmoniums. An nth-order harmonium is an n-tuple of random variables, where the conditional distribution of each variable given all the others is always an element of the same exponential family. I show that harmoniums have a recursive structure which allows them to be analyzed at coarser and finer levels of detail. - I define a class of harmoniums called rectified harmoniums, which are constrained to have priors which are conjugate to their posteriors. As a consequence of this, rectified harmoniums afford efficient sampling and learning. - I develop deep harmoniums, which are harmoniums which can be represented by hierarchical, undirected graphs. I develop the theory of rectification for deep harmoniums, and develop a novel algorithm for training deep generative models. - I show how to implement a variety of optimal and near-optimal Bayes filters by combining the solution to Bayes' rule provided by rectified harmoniums, with predictions computed by a recurrent neural network. I then show how to train a neural network to implement Bayesian filtering when the transition and emission distributions are unknown. - I show how some well-established models of neural activity are special cases of the theory I present in this dissertation, and how these models can be generalized with the theory of rectification. - I show how the theory that I present can model several neural phenomena including proprioception and gain-field modulation of tuning curves. - I introduce a library for the programming language Haskell, within which I have implemented all the simulations presented in this dissertation. This library uses concepts from Riemannian geometry to provide a rigorous and efficient environment for implementing complex numerical simulations. I also use the results presented in this dissertation to argue for the fundamental role of neural computation in embodied cognition. I argue, in other words, that before we will be able to build truly intelligent robots, we will need to truly understand biological brains.

Missed hospital appointments is a globally acknowledged problem. In order to minimize the cost associated with this, attempts have been made to predict what appointments will be missed using statistical models and machine learning. However, in all previous models, information about a patient’s previous appointments has been ignored to some extent. In this thesis, a novel way of incorporating previous appointment data in more detail is proposed. This is done by firstly estimating a prior attendance probability based on general data using an artificial neural network, and then updating it using Bayesian inference. In the updating process, the information about a patient’s previous appointments is used in order to capture eventual unique patterns and behavior. This is done by weighting the outcome of the past appointments according to how similar those appointments are to the appointment that is to be predicted. Additionally, different ways of measuring the uncertainty of predictions are evaluated. The results show that weighting the outcome of previous appointments differently improves the performance of the predictions, which indicates that the proposed model manages to capture patients’ individual patterns. This improvement is apparent regardless of what model used to estimate the prior attendance probability. Furthermore, the uncertainty measurements correlated well with incorrect predictions, suggesting that they can be used to determine the reliability of a prediction.

Year after years, the amount of data that we continuously generate is increasing. When this situation started the main challenge was to find a way to store the huge quantity of information. Nowadays, with the increasing availability of storage facilities, this problem is solved but it gives us a new issue to deal with: find tools that allow us to learn from this large data sets. In this thesis, a framework for Bayesian learning with the ability to scale to large data sets is studied. We present the Stochastic Gradient Langevin Dynamics (SGLD) framework and show that in some cases its approximation of the posterior distribution is quite poor. A reason for this can be that SGLD estimates the gradient of the log-likelihood with a high variability due to naïve sampling. Our approach combines accurate proxies for the gradient of the log-likelihood with SGLD. We show that it produces better results in terms of convergence to the correct posterior distribution than the standard SGLD, since accurate proxies dramatically reduce the variance of the gradient estimator. Moreover, we demonstrate that this approach is more efficient than the standard Markov Chain Monte Carlo (MCMC) method and that it exceeds other techniques of variance reduction proposed in the literature such as SAGA-LD algorithm. This approach also uses control variates to improve SGLD so that it is straightforward the comparison with our approach. We apply the method to the Logistic Regression model.

Chaos theory is a branch of modern mathematics concerning the non-linear dynamic systems that are highly sensitive to their initial states. It has extensive real-world applications, such as weather forecasting and stock market prediction. The Lorenz system, defined by three ordinary differential equations (ODEs), is one of the simplest and most popular chaotic models. Historically research has focused on understanding the Lorenz system's mathematical characteristics and dynamical evolution including the inherent chaotic features it possesses. In this thesis, we take a data-driven approach and propose the task of predicting future states of the chaotic system from limited observations. We explore two directions, answering two distinct fundamental questions of the system based on how informed we are about the underlying model. When we know the data is generated by the Lorenz System with unknown parameters, our task becomes parameter estimation (a white-box problem), or the ``inverse'' problem. When we know nothing about the underlying model (a black-box problem), our task becomes sequence prediction. We propose two algorithms for the white-box problem: Markov-Chain-Monte-Carlo (MCMC) and a Multi-Layer-Perceptron (MLP). Specially, we propose to use the Metropolis-Hastings (MH) algorithm with an additional random walk to avoid the sampler being trapped into local energy wells. The MH algorithm achieves moderate success in predicting the $\rho$ value from the data, but fails at the other two parameters. Our simple MLP model is able to attain high accuracy in terms of the $l_2$ distance between the prediction and ground truth for $\rho$ as well, but also fails to converge satisfactorily for the remaining parameters. We use a Recurrent Neural Network (RNN) to tackle the black-box problem. We implement and experiment with several RNN architectures including Elman RNN, LSTM, and GRU and demonstrate the relative strengths and weaknesses of each of these methods. Our results demonstrate the promising role of machine learning and modern statistical data science methods in the study of chaotic dynamic systems. The code for all of our experiments can be found on \url{https://github.com/Yajing-Zhao/}

<div><div><div><p>Structural identification is a useful tool for detecting damage and damage evolution in a structure. The initiation of damage in a structure and its subsequent growth are mainly associated with nonlinear behaviors. While linear dynamics of a structure are easy to simulate, nonlinear structural dynamics have more complex dynamics and amplitude dependence that do require more sophisticated simulation tools and identification methods compared to linear systems. Additionally, there are generally many more parameters in nonlinear models and the responses may not be sensitive to all of them for all inputs. To develop model selection methods, an experiment is conducted that uses an existing device with repeatable behavior and having an expected model from the literature. In this case, an MR damper is selected as the experimental device. The objective of this research is to develop and demonstrate a method to select the most appropriate model from a set of identified stochastic models of a nonlinear device. The method is developed using numerical example of a common nonlinear system, and is then implemented on an experimental structural system with unknown nonlinear properties. Bayesian methods are used because they provide a distinct advantage over many other existing methods due to their ability to provide confidence on answers given the observed data and initial uncertainty. These methods generate a description of the parameters of the system given a set of observations. First, the selected model of the MR damper is simulated and used for demonstrating the results on a numerical example. Second, the model selection process is demonstrated on an experimental structure based on experimental data. This study explores the use of the Bayesian approach for nonlinear structural identification and identifies a number of lessons for others aiming to employ Bayesian inference.</p></div></div></div>