Data generation has vastly exceeded manual annotation in several areas of astronomy, biology, economy, geology, medicine and physics. At the same time, a public community of experts and hobbyists has developed around some of these disciplines thanks to open, editable web resources such as wikis and public annotation challenges. In this thesis I investigate under which conditions a combination of collaborative and automated curation could complete annotation tasks unattainable by human curators alone. My exemplar curation process is taken from the molecular biology domain: the association all existing enzymes (proteins catalysing a chemical reaction) with their function. Assigning enzymatic function to the proteins in a genome is the first essential problem of metabolic reconstruction, important for biology, medicine, industrial production and environmental studies. In the protein database UniProt, only 3% of the records are currently manually curated and only 60% of the 17 million recorded proteins have some functional annotation, including enzymatic annotation. The proteins in UniProt represent only about 380,000 animal species (2,000 of which have completely sequenced genomes) out of the estimated millions of species existing on earth. The enzyme annotation task already applies to millions of entries and this number is bound to increase rapidly as sequencing efforts intensify. To guide my analysis I first develop a basic model of collaborative curation and evaluate it against molecular biology knowledge bases. The analysis highlights a surprising similarity between open and closed annotation environments on metrics usually connected with “democracy” of content. I then develop and evaluate a method to enhance enzyme function annotation using machine learning which demonstrates very high accuracy, recall and precision and the capacity to scale to millions of enzyme instances. This method needs only a protein sequence as input and is thus widely applicable to genomic and metagenomic analysis. The last phase of the work uses active and guided learning to bring together collaborative and automatic curation. In active learning a machine learning algorithm suggests to the human curators which entry should be annotated next. This strategy has the potential to coordinate and reduce the amount of manual curation while improving classification performance and reducing the number of training instances needed. This work demonstrates the benefits of combining classic machine learning and guided learning to improve the quantity and quality of enzymatic knowledge and to bring us closer to the goal of annotating all existing enzymes.

There has been growing recent interest in the field of active learning for binary classification. This thesis develops a Bayesian approach to active learning which aims to minimise the objective function on which the learner is evaluated, namely the expected misclassification cost. We call this approach the expected cost reduction approach to active learning. In this form of active learning queries are selected by performing a `lookahead' to evaluate the associated expected misclassification cost. \paragraph{} Firstly, we introduce the concept of a \textit{query density} to explicitly model how new data is sampled. An expected cost reduction framework for active learning is then developed which allows the learner to sample data according to arbitrary query densities. The model makes no assumption of independence between queries, instead updating model parameters on the basis of both which observations were made \textsl{and} how they were sampled. This approach is demonstrated on the probabilistic high-low game which is a non-separable extension of the high-low game presented by \cite{Seung_etal1993}. The results indicate that the Bayes expected cost reduction approach performs significantly better than passive learning even when there is considerable overlap between the class distributions, covering $30\%$ of input space. For the probabilistic high-low game however narrow queries appear to consistently outperform wide queries. We therefore conclude the first part of the thesis by investigating whether or not this is always the case, demonstrating examples where sampling broadly is favourable to a single input query. \paragraph{} Secondly, we explore the Bayesian expected cost reduction approach to active learning within the pool-based setting. This is where learning is limited to a finite pool of unlabelled observations from which the learner may select observations to be queried for class-labels. Our implementation of this approach uses Gaussian process classification with the expectation propagation approximation to make the necessary inferences. The implementation is demonstrated on six benchmark data sets and again demonstrates superior performance to passive learning.

Diagnosis is the first analysis step for uncovering the root cause of failure for a defective chip. It is a fast and non-destructive approach to preliminarily identify and locate possible defects in a failing chip. Despite many advances in diagnosis techniques, it is often the case, however, that resolution, i.e., the number of locations or candidates reported by diagnosis, exceeds the number of actual failing locations. To address this major challenge, a novel, machine-learning-based resolution improvement methodology named PADRE (Physically-Aware Diagnostic Resolution Enhancement) is described. PADRE uses easily-available tester and simulation data to extract features that uniquely characterize each candidate. PADRE applies machine learning to the features to identify candidates that correspond to the actual failure locations. Through various experiments, PADRE is shown to significantly improve resolution with virtually no negative impact on accuracy. Specifically, in simulation experiments, the number of defects that have perfect resolution is increased by 5x with little degradation of accuracy. An important investigation that typically follows diagnosis is Physical Failure Analysis (PFA), which can also provide information that is helpful for improving diagnosis. PADRE influences PFA within a novel, active learning (AL) based PFA selection approach. An active-learning based PADRE (AL PADRE) selects the most useful defects for PFA in order to improve diagnostic resolution. Experiments show AL PADRE can reach an accuracy of 90% with 60% less PFA, on average, compared to conventional defect selection for PFA. In addition, during the yield learning process, the failing mechanisms that lead to defective chips may change due to perturbations in the fabrication process. It is important for PADRE to perform robustly through the entire yield learning process. Therefore, additional techniques are developed to monitor the effectiveness of PADRE in real time, as well as to update PADRE efficiently and stably to cope with changing failure mechanisms.

The concept of flipped learning has received significant attention in recent years. In a flipped learning methodology, students view instructional videos and complete related assignments before class, so that face-to-face time with the instructor can be spent applying the knowledge and skills they were introduced to in the pre-class assignments. This study aims to determine the effectiveness of a flipped learning method for teaching information literacy (IL) skills to undergraduate students compared to a traditional teaching method where the majority of face-to-face time is spent instructing. To evaluate this, a mixed methods research design was used, where results from qualitative interviews helped explain findings from test data, assignment completion data, and major paper rubric data. The IL tests resulted in a small but insignificant test score improvement for flipped participants. Interviewed flipped participants reported mainly positive feelings toward flipped learning, whereas all flipped ESL interviewees related strong positive feedback towards flipped learning. / Graduate / 0515 / 0727 / 0710 / rmccue@uvic.ca

In multi-agent Markov Decision Processes, a controllable agent must perform optimal planning in a dynamic and uncertain environment that includes another unknown and uncontrollable agent. Given a task specification for the controllable agent, its ability to complete the task can be impeded by an inaccurate model of the intent and behaviors of other agents. In this work, we introduce an active policy inference algorithm that allows a controllable agent to infer a policy of the environmental agent through interaction. Active policy inference is data-efficient and is particularly useful when data are time-consuming or costly to obtain. The controllable agent synthesizes an exploration-exploitation policy that incorporates the knowledge learned about the environment's behavior. Whenever possible, the agent also tries to elicit behavior from the other agent to improve the accuracy of the environmental model. This is done by mapping the uncertainty in the environmental model to a bonus reward, which helps elicit the most informative exploration, and allows the controllable agent to return to its main task as fast as possible. Experiments demonstrate the improved sample efficiency of active learning and the convergence of the policy for the controllable agents.

The labeling of language resources is a time consuming task, whether aided by machine learning or not. Much of the prior work in this area has focused on accelerating human annotation in the context of machine learning, yielding a variety of active learning approaches. Most of these attempt to lead an annotator to label the items which are most likely to improve the quality of an automated, machine learning-based model. These active learning approaches seek to understand the effect of item selection on the machine learning model, but give significantly less emphasis to the effect of item selection on the human annotator. In this work, we consider a sentiment labeling task where existing, traditional active learning seems to have little or no value. We focus instead on the human annotator by ordering the items for better annotator efficiency.

Navigating through the debris of the information explosion requires powerful, flexible search tools. These tools must be both useful and useable; that is, they must do their jobs effectively without placing too many burdens on the user. While general interest search engines, such as Google, have addressed this latter challenge well, more topic-specific search engines, such as PubMed, have not. These search engines, though effective, often require training in their use, as well as in-depth knowledge of the domain over which they operate. Furthermore, search results are often returned in an order irrespective of users' preferences, forcing them to manually search through search results in order to find the documents they find most useful. To solve these problems, we intend to learn ranking functions from user relevance preferences. Applying these ranking functions to search results allows us to improve search usability without having to reengineer existing, effective search engines. Using ranking SVMs and active learning techniques, we can effectively learn what is relevant to a user from relatively small amounts of preference data, and apply these learned models as ranking functions. This gives users the convenience of seeing relevance-ordered search results, which are tailored to their preferences as opposed to using a one-size-fits-all sorting method. As giving preference feedback does not require in-depth domain knowledge, this approach is suitable for use by domain experts as well as neophytes. Furthermore, giving preference feedback does not require a great deal of training, adding very little overhead to the search process.

No description available.

We discuss a formulation for active example selection for function learning problems. This formulation is obtained by adapting Fedorov's optimal experiment design to the learning problem. We specifically show how to analytically derive example selection algorithms for certain well defined function classes. We then explore the behavior and sample complexity of such active learning algorithms. Finally, we view object detection as a special case of function learning and show how our formulation reduces to a useful heuristic to choose examples to reduce the generalization error.

The purpose of this study is to investigate the teacher change process and extend our understanding of how variability in the ways that primary school teachers in Bangladesh implement innovative pedagogical practices, such as active learning, reflects variations in their understanding, attitude, experience, and skill in the use of those pedagogical approaches. Multiple forms of data gathering were employed based on the concerns-based adoption model (CBAM) including an open-ended statement of concern, interviews, and class observations from a purposive sample of ten teachers working in ten different schools. Additional interviews were also conducted with staff responsible for the teachers' professional development. Five main findings emerged from the research. First, there was a split between novice teachers who were committed to following the prescriptive lesson plans and more experienced teachers who adapted their lessons to accommodate differences in student readiness and performance. Second, the majority of teachers appeared to be satisfied with their use active learning methods and the mandated lessons with little projected variation in how they will implement the innovation in the future. Third, the class observation findings indicate that the majority of teachers were rated as ideal users of active learning methods in the classroom. Fourth, findings indicate that professional development and a commitment to building networks among teachers and support staff helped facilitate teachers' confidence and competency. Fifth, among the most influential factors shaping teachers' use of active learning methods were the availability of supplementary learning and teaching resources. Implications for professional development and support for teachers, the applicability of CBAM-based research in low-income country contexts like Bangladesh, as well as future areas of comparative, international, and development education research are discussed in light of those findings.