System identification studies how to construct mathematical models for dynamical systems from the input and output data, which finds applications in many scenarios, such as predicting future output of the system or building model based controllers for regulating the output the system. Among many other methods, convex optimization is becoming an increasingly useful tool for solving system identification problems. The reason is that many identification problems can be formulated as, or transformed into convex optimization problems. This transformation is commonly referred to as the convexification technique. The first theme of the thesis is to understand the efficacy of the convexification idea by examining two specific examples. We first establish that a l1 norm based approach can indeed help in exploiting the sparsity information of the underlying parameter vector under certain persistent excitation assumptions. After that, we analyze how the nuclear norm minimization heuristic performs on a low-rank Hankel matrix completion problem. The underlying key is to construct the dual certificate based on the structure information that is available in the problem setting.         Recursive algorithms are ubiquitous in system identification. The second theme of the thesis is the study of some existing recursive algorithms, by establishing new connections, giving new insights or interpretations to them. We first establish a connection between a basic property of the convolution operator and the score function estimation. Based on this relationship, we show how certain recursive Bayesian algorithms can be exploited to estimate the score function for systems with intractable transition densities. We also provide a new derivation and interpretation of the recursive direct weight optimization method, by exploiting certain structural information that is present in the algorithm. Finally, we study how an improved randomization strategy can be found for the randomized Kaczmarz algorithm, and how the convergence rate of the classical Kaczmarz algorithm can be studied by the stability analysis of a related time varying linear dynamical system.

Thesis (PhD)--Stellenbosch University, 2014. / ENGLISH ABSTRACT: We investigate the procedure of assigning probabilities to logical statements. The simplest case is that of equilibrium statistical mechanics and its fundamental assumption of equally likely states. Rederiving the formulation led us to question the assumption of logical independence inherent to the construction and speci cally its inability to update probability when data becomes available. Consequently we replace the assumption of logical independence with De Finetti's concept of exchangeability. To use the corresponding representation theorems of De Finetti requires us to assign prior distributions for some general parameter spaces. We propose the use of stability properties to identify suitable prior distributions. The combination of exchangeable likelihoods and corresponding prior distributions results in more general evidence distribution assignments. These new evidence assignments generalise the Shannon entropy to other entropy measures. The goal of these entropy formulations is to provide a general framework for constructing models. / AFRIKAANSE OPSOMMING: Ons ondersoek the prosedure om waarskynlikhede aan logiese stellings toe te ken. Die eenvoudigste geval is die van ewewig-statistiese meganika en die ooreenkomstige fundamentele aanname van ewekansige toestande. Hera eiding van die standaard formulering lei ons tot die bevraagtekening van die aanname van logiese onafhanklikheid en spesi ek die onmoontlikheid van opdatering van waarskynlikheid wanneer data beskikbaar raak. Gevolglik vervang ons die aanname van logiese onafhanklikheid met De Finetti se aanname van omruilbaarheid. Om die ooreenkomstige voorstelling stellings te gebruik moet ons a priori verdelings konstrueer vir 'n paar algemene parameter-ruimtes. Ons stel voor dat stabiliteits-eienskappe gebruik moet word om geskikte a priori distribusies te identi seer. Die kombinase van omruilbare aanneemlikheids funksies en die ooreenkomstige a priori verdelings lei ons tot nuwe toekennings van getuienis-verdelings. Hierdie nuwe getuienesverdelings is n veralgemening van Shannon se entropie na ander entropie-maatstawwe. Die doel van hierdie entropie formalismes is om 'n raamwerk vir modelkonstruksie te verskaf.

Methods to detect patterns of variation associated with ongoing positive selection often focus on identifying regions of the genome with extended haplotype homozygosity - indicative of recently shared ancestry. Whilst these have been shown to be powerful they have two major challenges. First, these methods are constructed to detect variation associated with a classical selective sweep; a single haplotype background gets swept up to a higher than expected frequency given its age. Recently studies have shown that other forms of positive selection, e.g. selection on standing variation, may be more prevalent than previous thought. Under such evolution, a mutation that is already segregating in the population becomes beneficial, possibly as a result of an environmental change. The second challenge with these methods is that they base their inference on non-parametric tests of significance which can result in uncontrolled false positive rates. We tackle these problems using two approaches. First, by exploiting a widely used model in population genomics we construct a new approach to detect regions where a subset of the chromosomes are much more related than expected genome-wide. Using this metric we show that it is sensitive to both classical selective sweeps, and to soft selective sweeps, e.g. selection on standing variation. Second, building on existing methods, we construct a Bayesian test which bi-partitions chromosomes at every position based on their allelic type and tests for association between chromosomes carrying one allele and significantly reduced time to common ancestor. Using simulated data we show that this approach results in a powerful, fast, and robust approach to detect signals of positive selection in sequence data. Moreover by comparing our model to existing techniques we show that we have similar power to detect recent classical selective sweeps, and considerably greater power to detect soft selective sweeps. We apply our method, ABACUS, to three human populations using data from the 1000 Genome Project. Using existing and novel candidates of positive selection, we show that the results between ABACUS and existing methods are comparable in regions of classical selection, and are arguably superior in regions that show evidence for recent selection on standing variation.

Effective use of Machine Learning to support extracting maximal information from limited sensor data is one of the important research challenges in robotic sensing. This thesis develops techniques for detecting and characterizing patterns in noisy sensor data. Our Bayesian Aggregation (BA) algorithmic framework can leverage data fusion from multiple low Signal-To-Noise Ratio (SNR) sensor observations to boost the capability to detect and characterize the properties of a signal generating source or process of interest. We illustrate our research with application to the nuclear threat detection domain. Developed algorithms are applied to the problem of processing the large amounts of gamma ray spectroscopy data that can be produced in real-time by mobile radiation sensors. The thesis experimentally shows BA’s capability to boost sensor performance in detecting radiation sources of interest, even if the source is faint, partiallyoccluded, or enveloped in the noisy and variable radiation background characteristic of urban scenes. In addition, BA provides simultaneous inference of source parameters such as the source intensity or source type while detecting it. The thesis demonstrates this capability and also develops techniques to efficiently optimize these parameters over large possible setting spaces. Methods developed in this thesis are demonstrated both in simulation and in a radiation-sensing backpack that applies robotic localization techniques to enable indoor surveillance of radiation sources. The thesis further improves the BA algorithm’s capability to be robust under various detection scenarios. First, we augment BA with appropriate statistical models to improve estimation of signal components in low photon count detection, where the sensor may receive limited photon counts from either source and/or background. Second, we develop methods for online sensor reliability monitoring to create algorithms that are resilient to possible sensor faults in a data pipeline containing one or multiple sensors. Finally, we develop Retrospective BA, a variant of BA that allows reinterpretation of past sensor data in light of new information about percepts. These Retrospective capabilities include the use of Hidden Markov Models in BA to allow automatic correction of a sensor pipeline when sensor malfunction may be occur, an Anomaly- Match search strategy to efficiently optimize source hypotheses, and prototyping of a Multi-Modal Augmented PCA to more flexibly model background and nuisance source fluctuations in a dynamic environment.

The application of Bayesian networks within the field of institutional research is explored through the development of a Bayesian network used to predict first- to second-year retention of undergraduates. A hybrid approach to model development is employed, in which formal elicitation of subject-matter expertise is combined with machine learning in designing model structure and specification of model parameters. Subject-matter experts include two academic advisors at a small, private liberal arts college in the southeast, and the data used in machine learning include six years of historical student-related information (i.e., demographic, admissions, academic, and financial) on 1,438 first-year students. Netica 5.12, a software package designed for constructing Bayesian networks, is used for building and validating the model. Evaluation of the resulting model’s predictive capabilities is examined, as well as analyses of sensitivity, internal validity, and model complexity. Additionally, the utility of using Bayesian networks within institutional research and higher education is discussed. The importance of comprehensive evaluation is highlighted, due to the study’s inclusion of an unbalanced data set. Best practices and experiences with expert elicitation are also noted, including recommendations for use of formal elicitation frameworks and careful consideration of operating definitions. Academic preparation and financial need risk profile are identified as key variables related to retention, and the need for enhanced data collection surrounding such variables is also revealed. For example, the experts emphasize study skills as an important predictor of retention while noting the absence of collection of quantitative data related to measuring students’ study skills. Finally, the importance and value of the model development process is stressed, as stakeholders are required to articulate, define, discuss, and evaluate model components, assumptions, and results.

This dissertation describes health care delivery in a Ryan White Program (RWP) HIV clinic, with a focus on medical home care, using the Bayesian Case Study Method (BCSM). The RWP funds medical care for uninsured HIV patients and Pappas and colleagues (2014) suggested enhanced HIV care build upon medical home models of care rooted in the RWP. However, little research describes how RWP clinics operate as medical homes. This study developed the BCSM to describe medical home care at a RWP clinic. The BCSM combines a case study framework with Bayesian statistics for a novel approach to mixed method, descriptive studies. Roberts (2002) and Voils (2009) used mixed-method Bayesian approaches and this dissertation contributes to this work. For this study, clinic staff and patients participated in interviews and surveys. I used Bayes’ Theorem to combine interview data, by use of subjective priors, with survey data to produce Bayesian posterior means that indicate the extent to which medical home care was provided. Subjective priors facilitate the inclusion of valuable stakeholder belief in posteriors. Using the BCSM, posterior means succinctly describe qualitative and quantitative data, in a way other methods of mixing data do not, which is useful for decision makers. Posterior means indicated that coordinated, comprehensive, and ongoing care was provided at the clinic; however, accessible care means were lower reflecting an area in need of improvement. Interview data collected for subjective priors captured detailed service delivery descriptions. For example, interview data described how medical and support services were coordinated and highlighted the role of social determinants of health (SDH). Namely, coordinated and comprehensive services that addressed SDH, such as access to housing, food, and transportation, were necessary for patients to focus on their HIV and utilize healthcare. This case study addressed a gap in the literature regarding descriptions of how RWP clinics provide medical home care. For domains with high posterior means, the associated interview data can be used to plan HIV care in non-RWP settings. Future research should describe other RWP HIV medical homes so this information can be used to plan enhanced HIV care across the healthcare system.

As a central tenet of falsificationism, Karl Popper holds that all possible scientific theories individually have a probability equal to zero. Popper’s position rests upon the Principle of Indifference, the equiprobability of mutually exclusive outcomes, to derive this zero probability. In this paper, I will illustrate that the Principle of Indifference fails to compute objective probabilities in cases in which an epistemic agent faces ignorance. Prior to experience, there is no sufficient reason to prefer any probability distribution to any other; yet, the Principle of Indifference implies a preference for a uniform probability distribution. Distribution preference is determined by the relevant experience and rational expectations of epistemic agents. Relevant experience is defined by observations and other sense experience regarding the relevant trial. Rational expectations represents the non-arbitrarity of distribution preference. Without rational expectations, the distribution preference is arbitrary even when informed by experience. If an agent lacks relevant experience, then any distribution preference is arbitrary; however, if an agent possesses relevant experience, then the Principle of Indifference does not apply. A rejection of the Principle of Indifference undermines the necessity of zero probabilities for scientific theories in which case Popper’s conclusions of falsificationism do not follow. Objective probability, then, understood within the logical interpretation, is a problematic notion.

Identifying appropriate international distributors for small and medium-sized enterprises (SMEs) in the software industry for overseas markets can determine a firm's future endeavors in international expansion. SMEs lack the complex skills in market research and decision analysis to identify suitable partners to engage in global market entry. Foreign distributors hold responsibility to represent, sell, market, and add value to the software manufacturer's products in local markets. Identifying appropriate attributes to determine a suitable distributor is essential in assuring success in new export markets. Methods for partner selection have been addressed in the international marketing and information systems literature. Building on this literature, this dissertation develops an improved method for identifying suitable distributors in the SME software industry. The partner selection conundrum is modeled as a binary classification problem in that it involves predicting whether an alliance relationship will survive over a specific period. The challenge presented to researchers is not just the large number of variables involved in the selection process but also the inherent uncertainty in the decision making process. This study uses a Bayesian methodology for this classification task. A Naïve Bayes (NB) classification model was developed factoring sixteen alliance attributes identified in the partner selection literature and validated by domain experts who scored the importance of these attributes. Thirty years of partnership data that contributed to relationship longevity trained the model and held-back data was used to validate the model. The NB classification model returned accurate predictions in both the group of foreign distributors that succeeded and failed to reach the relationship longevity threshold of five years. The study's contribution to the software SME business community and its practitioners was the identification of an improved methodology for predictive success. The approach employed a simple Bayesian prediction model utilizing key alliance attributes to help software SMEs identify potential foreign distributor partners who can sustain relationship longevity from which to build a strong business partnership. Keeping the methodology simple is critical for SMEs who struggle with an abundance of challenges to maintain their corporate viability in the market place.

Discovering a word’s part of speech is an essential step in acquiring the grammar of a language. In this thesis we examine a variety of computational Bayesian models that use linguistic input available to children, in the form of transcribed child directed speech, to learn part of speech categories. Part of speech categories are characterised by contextual (distributional/syntactic) and word-internal (morphological) similarity. In this thesis, we assume language learners will be aware of these types of cues, and investigate exactly how they can make use of them. Firstly, we enrich the context of a standard model (the Bayesian Hidden Markov Model) by adding sentence type to the wider distributional context.We show that children are exposed to a much more diverse set of sentence types than evident in standard corpora used for NLP tasks, and previous work suggests that they are aware of the differences between sentence type as signalled by prosody and pragmatics. Sentence type affects local context distributions, and as such can be informative when relying on local context for categorisation. Adding sentence types to the model improves performance, depending on how it is integrated into our models. We discuss how to incorporate novel features into the model structure we use in a flexible manner, and present a second model type that learns to use sentence type as a distinguishing cue only when it is informative. Secondly, we add a model of morphological segmentation to the part of speech categorisation model, in order to model joint learning of syntactic categories and morphology. These two tasks are closely linked: categorising words into syntactic categories is aided by morphological information, and finding morphological patterns in words is aided by knowing the syntactic categories of those words. In our joint model, we find improved performance vis-a-vis single-task baselines, but the nature of the improvement depends on the morphological typology of the language being modelled. This is the first token-based joint model of unsupervised morphology and part of speech category learning of which we are aware.

Wrapper feature selection approaches are widely used to select a small subset of relevant features from a dataset. However, Wrappers suffer from the fact that they only use a single classifier when selecting the features. The problem of using a single classifier is that each classifier is of a different nature and will have its own biases. This means that each classifier will select different feature subsets. To address this problem, this thesis aims to investigate the effects of using different classifiers for Wrapper feature selection. More specifically, it aims to investigate the effects of using different number of classifiers and classifiers of different nature. This aim is achieved by proposing a new data mining method called Wrapper-based Decision Trees (WDT). The WDT method has the ability to combine multiple classifiers from four different families, including Bayesian Network, Decision Tree, Nearest Neighbour and Support Vector Machine, to select relevant features and visualise the relationships among the selected features using decision trees. Specifically, the WDT method is applied to investigate three research questions of this thesis: (1) the effects of number of classifiers on feature selection results; (2) the effects of nature of classifiers on feature selection results; and (3) which of the two (i.e., number or nature of classifiers) has more of an effect on feature selection results. Two types of user preference datasets derived from Human-Computer Interaction (HCI) are used with WDT to assist in answering these three research questions. The results from the investigation revealed that the number of classifiers and nature of classifiers greatly affect feature selection results. In terms of number of classifiers, the results showed that few classifiers selected many relevant features whereas many classifiers selected few relevant features. In addition, it was found that using three classifiers resulted in highly accurate feature subsets. In terms of nature of classifiers, it was showed that Decision Tree, Bayesian Network and Nearest Neighbour classifiers caused signficant differences in both the number of features selected and the accuracy levels of the features. A comparison of results regarding number of classifiers and nature of classifiers revealed that the former has more of an effect on feature selection than the latter. The thesis makes contributions to three communities: data mining, feature selection, and HCI. For the data mining community, this thesis proposes a new method called WDT which integrates the use of multiple classifiers for feature selection and decision trees to effectively select and visualise the most relevant features within a dataset. For the feature selection community, the results of this thesis have showed that the number of classifiers and nature of classifiers can truly affect the feature selection process. The results and suggestions based on the results can provide useful insight about classifiers when performing feature selection. For the HCI community, this thesis has showed the usefulness of feature selection for identifying a small number of highly relevant features for determining the preferences of different users.