With sensor technology gaining maturity and becoming ubiquitous, we are experiencing an unprecedented wealth of sensor data. In most sensing scenarios, the measurements generated by sensor networks are noisy and usually annotated with some measure of uncertainty. The problem we address in this thesis is how to estimate the accuracy of the sensor systems based on the probabilistic measurements they provide. This problem is increasingly common in many settings, such as multiple sensing services are competing for the same group of users, detecting faults in large scale networks, or establishing trustworthiness of different individuals in social sensing. It is also challenging in many ways, for instance, the ground truth of the monitored states is absent, the users often lack a clear view of the implementation details of the sensor systems, and the reported accuracy can be misleading. To address theses challenges, in this thesis we formulate the problem of estimating the accuracy of sensor systems in a general manner that applies to a broad spectrum of sensing scenarios. We then propose an accuracy estimation framework that breaks the problem into layers, which can be implemented in different ways. We present a novel inference-based accuracy estimation approach, which assesses the accuracy of sensor systems by comparing the reported measurements with the states inferred with the probabilistic measurements from all systems and available prior knowledge. We also propose a new learning-based approach for accuracy estimation, which employs novel parameter learning techniques. The learned parameters are either used to improve estimating the accuracy of sensor measurements, or to derive the accuracy of sensor systems directly in certain cases. We perform a systematic experimental evaluation on two datasets collected from real-world sensor deployments, where an array of different approaches are juxtaposed and compared extensively. We discuss how they trade accuracy for computation cost, and how this trade-off largely depends on the knowledge of the sensing scenarios. We also show that the proposed approaches outperform the competing ones in estimating accuracy and ranking the sensor systems.

Doctor of Philosophy / In my research, I developed protein model quality estimation methods aimed at evaluating the reliability of computationally predicted protein models in the absence of experimentally solved ground truth structures. These methods specifically focus on estimating errors within the protein models to quantify their structural accuracy. Recognizing that even the most advanced protein structure prediction techniques may produce models with errors, I also developed a complementary protein model refinement method. This refinement method iteratively optimizes the weakly modeled regions, guided by the error estimation module of my quality estimation approach. The development of these model quality estimation methods, therefore, not only offers valuable insights into the structural reliability of protein models but also contributes to optimizing the overall reliability of protein models generated by state-of-the-art computational methods.

Background: In a computed protein multiple sequence alignment, the coreness of a column is the fraction of its substitutions that are in so-called core columns of the gold-standard reference alignment of its proteins. In benchmark suites of protein reference alignments, the core columns of the reference alignment are those that can be confidently labeled as correct, usually due to all residues in the column being sufficiently close in the spatial superposition of the known three-dimensional structures of the proteins. Typically the accuracy of a protein multiple sequence alignment that has been computed for a benchmark is only measured with respect to the core columns of the reference alignment. When computing an alignment in practice, however, a reference alignment is not known, so the coreness of its columns can only be predicted. Results: We develop for the first time a predictor of column coreness for protein multiple sequence alignments. This allows us to predict which columns of a computed alignment are core, and hence better estimate the alignment's accuracy. Our approach to predicting coreness is similar to nearest-neighbor classification from machine learning, except we transform nearest-neighbor distances into a coreness prediction via a regression function, and we learn an appropriate distance function through a new optimization formulation that solves a large-scale linear programming problem. We apply our coreness predictor to parameter advising, the task of choosing parameter values for an aligner's scoring function to obtain a more accurate alignment of a specific set of sequences. We show that for this task, our predictor strongly outperforms other column-confidence estimators from the literature, and affords a substantial boost in alignment accuracy.

This study uses a combination of user testing and analysis to evaluate the impact of brush and tooltip on the comprehension of time series visualizations. Employing a sequential mixed-methods approach, with qualitative data from semi-structured interviews used to inform the design of a visualization tool, followed by a quantitative user study to validate it. Sixteen (16) participants from various fields of study, predominantly computer science, participated in the study. A MANOVA test was conducted with results indicating a significant statistical difference between the groups. Results deriving from the study show that the use of brush and tooltip increases user accuracy on detecting outliers, as for perception of trends and patterns. The study’s context was limited to desktop usage, and all participants were treated as a homogenous group, presenting potential limitations in applying these findings to other devices or more diverse user groups. The results provide information about improving time series data visualizations for facilitating more efficient and effective understanding, which can be relevant specifically to data analysts and academic researchers.

This thesis describes a combination of two current areas of research: the Crayons image classifier system and active learning. Currently Crayons provides no guidance to the user in what pixels should be labeled or when the task is complete. This work focuses on two main areas: 1) active learning for user guidance, and 2) accuracy estimation as a measure of completion. First, I provide a study through simulation and user experiments of seven active learning techniques as they relate to Crayons. Three of these techniques were specifically designed for use in Crayons. These three perform comparably to the others and are much less computationally intensive. A new widget is proposed for use in the Crayons environment giving an overview of the system "confusion". Second, I give a comparison of four accuracy estimation techniques relating to true accuracy and for use as a completion estimate. I show how three traditional accuracy estimation techniques are ineffective when placed in the Crayons environment. The fourth technique uses the same computation as the three new active learning techniques proposed in this work and thus requires little extra computation and outstrips the other three as a completion estimate both in simulation and user experiments.

One of the major applications of wireless sensor networks is to sense accurate and reliable data from the physical environment with or without a priori knowledge of data statistics. To extract accurate data from the physical environment, we investigate spatial data correlation among sensor nodes to develop data accuracy models. We propose three data accuracy models namely Estimated Data Accuracy (EDA) model, Cluster based Data Accuracy (CDA) model and Distributed Cluster based Data Accuracy (DCDA) model with a priori knowledge of data statistics. Due to the deployment of high density of sensor nodes, observed data are highly correlated among sensor nodes which form distributed clusters in space. We describe two clustering algorithms called Deterministic Distributed Clustering (DDC) algorithm and Spatial Data Correlation based Distributed Clustering (SDCDC) algorithm implemented under CDA model and DCDA model respectively. Moreover, due to data correlation in the network, it has redundancy in data collected by sensor nodes. Hence, it is not necessary for all sensor nodes to transmit their highly correlated data to the central node (sink node or cluster head node). Even an optimal set of sensor nodes are capable of measuring accurate data and transmitting the accurate, precise data to the central node. This reduces data redundancy, energy consumption and data transmission cost to increase the lifetime of sensor networks. Finally, we propose a fourth accuracy model called Adaptive Data Accuracy (ADA) model that doesn't require any a priori knowledge of data statistics. ADA model can sense continuous data stream at regular time intervals to estimate accurate data from the environment and select an optimal set of sensor nodes for data transmission to the network. Data transmission can be further reduced for these optimal sensor nodes by transmitting a subset of sensor data using a methodology called Spatio-Temporal Data Prediction (STDP) model under data reduction strategies. Furthermore, we implement data accuracy model when the network is under a threat of malicious attack.