Increasingly, predictive models are used to support routine business de- cisions and are integral to the strategic competitive business strategies for a wide range of industries. Most often, data-driven predictive models are in- duced from training data obtained through the businesss routine operations. However, recent research on policies for intelligent information acquisitions suggests that proactive acquisition of information can improve models at a lower cost. Most active information acquisition policies are accuracy centric; they aim to identify acquisitions of training data that are particularly benefi- cial for improving the predictive accuracy of a given model. In practice, however, inferences from a predictive model are often used along with inferences from other predictive models as well as constant factors to inform arbitrarily complex decisions. In this dissertation, I discuss how these settings motivate a new kind of collaborative information acquisition (CIA) policies that exploit knowledge of the decision to allow multiple predictive models to collaboratively prioritize the prospective information acquisitions, so as to best improve the decisions they inform jointly. I present a framework for CIA policies and two specific CIA policies: CIA for binary decisions (CIA-BD), and CIA for top-ranked opportu- nities in terms of expected revenue (CIA-TR). Extensive empirical evaluations of the policies on real-world data suggest that the notion of CIA policies is indeed a valuable one. In particular, I demonstrate that these two new poli- cies lead to superior decision-making performances as compared to those of alternative policies that are either decision-centric or do not allow multiple models to collaboratively prioritize acquisitions. The performance exhibited by the CIA policies suggest that these policies are able to effectively exploit knowledge of the decisions to avoid greedy improvements in accuracy of any individual model informing the decisions; instead, they promote improvements in any one or all of the models when such improvements are likely to benefit the decisions. / text

As complex systems become more available, the possibility to leverage human intelligence to continuously train these systems is becoming increasingly valuable. Collecting and incorporating feedback from end-users into the system development processes could hold great potential for future development of autonomous systems, but it is not without difficulties A literature review was conducted with the aim to review and help categorize the different dynamics relevant to the act of collecting and implementing user feedback in system development processes. Practical examples of such system are commonly found in active and interactive learning systems, which were studied with a particular interest towards possible novel applications in the industrial sector. This review was complimented by an exploratory experiment, aimed at testing how system accuracy affected the feedback provided by users for a simulated people recognition system. The findings from these studies indicate that when and how feedback is given along with the context of use is of importance for the interplay between system and user. The findings are discussed in relation to current directions in machine learning and interactive learning systems. The study concludes that factors such as system criticality, the phase in which feedback is given, how feedback is given, and the user’s understanding of the learning process all have a large impact on the interactions and outcomes of the user-automation interplay. Suggestions of how to design feedback collection for increased user engagement and increased data assimilation are given.