Intelligence analysts today are faced with many challenges, chief among them being the need to fuse disparate streams of data, as well as rapidly arrive at analytical decisions and quantitative predictions for use by policy makers. These problems are further exacerbated by the sheer volume of data that is available to intelligence analysts. Machine learning methods enable the automated transduction of such large datasets from raw feeds to actionable knowledge but successful use of such methods require integrated frameworks for contextualizing them within the work processes of the analyst. Intelligence analysts typically distinguish between three classes of problems: collections, analysis, and operations. This dissertation specifically focuses on two problems in analysis: i) the reconstruction of shredded documents using a visual analytic framework combining computer vision techniques and user input, and ii) the design and implementation of a system for event forecasting which allows an analyst to not just consume forecasts of significant societal events but also understand the rationale behind these alerts and the use of data ablation techniques to determine the strength of conclusions. This work does not attempt to replace the role of the analyst with machine learning but instead outlines several methods to augment the analyst with machine learning. In doing so this dissertation also explores the responsibilities of an analyst in evaluating complex models and decisions made by these models. Finally, this dissertation defines a list of responsibilities for models designed to aid the analyst's work in evaluating and verifying the models. / Ph. D.

Intelligence analysts today are faced with many challenges, chief among them being the need to fuse disparate streams of data and rapidly arrive at analytical decisions and quantitative predictions for use by policy makers. A forecasting tool to anticipate key events of interest is an invaluable aid in helping analysts cut through the chatter. We present the design of user interfaces for the EMBERS system, an anticipatory intelligence system that ingests myriad open source data streams (e.g., news, blogs, tweets, economic and financial indicators, search trends) to generate forecasts of significant societal-level events such as disease outbreaks, protests, and elections. A key research issue in EMBERS is not just to generate high-quality forecasts but provide interfaces for analysts so they can understand the rationale behind these forecasts and pose why, what-if, and other exploratory questions. This thesis presents the design and implementation of three visualization interfaces for EMBERS. First, we illustrate how the rationale behind forecasts can be presented to users through the use of an audit trail and its associated visualization. The audit trail enables an analyst to drill-down from a final forecast down to the raw (and processed) data sources that contributed to the forecast. Second, we present a forensics tool called Reverse OSI that enables analysts to investigate if there was additional information either in existing or new data sources that can be used to improve forecasting. Unlike the audit trail which captures the transduction of data from raw feeds into alerts, Reverse OSI enables us to posit connections from (missed) forecasts back to raw feeds. Finally, we present an interactive machine learning approach for analysts to steer the construction of machine learning mod-els. This provides fine-grained control into tuning tradeoffs underlying EMBERS. Together, these three interfaces support a range of functionality in EMBERS, from visualization of algorithm output to a complete framework for user feedback via a tight human-algorithm loop. They are currently being utilized by a range of user groups in EMBERS: analysts, social scientists, and machine learning developers, respectively. / Master of Science

This treatise represents a descriptive study of the intelligence community in the United States. It explores the ramifications of terrorism on the intelligence function, post September 11, 2001. In-depth discussions concerning the structure of the U.S. intelligence community are presented as well as a focus on the defined steps of the intelligence process: planning and directions, collection, analysis, production, and dissemination. The final aspect of this study poses questions and issues relating to the restructuring of the U.S. intelligence community in light of the Homeland Security Act of 2002.

xv, 178 p. : ill. A print copy of this title is available through the UO Libraries under the call number: KNIGHT HM1101 .D54 2007 / The main goal of many political and intelligence forecasts is to effectively communicate risk information to decision makers (i.e. consumers). Standard reporting most often consists of a narrative discussion of relevant evidence concerning a threat, and rarely involves numerical estimates of uncertainty (e.g. a 5% chance). It is argued that numerical estimates of uncertainty will lead to more accurate representations of risk and improved decision making on the part of intelligence consumers. Little work has focused on how well consumers understand and use forecasts that include numerical estimates of uncertainty. Participants were presented with simulated intelligence forecasts describing potential terrorist attacks. These forecasts consisted of a narrative summary of the evidence related to the attack and numerical estimates of likelihood and potential harm. The primary goals were to explore how the structure of the narrative summary, the format of likelihood information, and the numerical ability (numeracy) of consumers affected perceptions of intelligence forecasts. Consumers perceived forecasts with numerical estimates of likelihood and potential harm as more useful than forecasts with only a narrative evidence summary. However, consumer's risk and likelihood perceptions were more greatly affected by the narrative evidence summary than the stated likelihood information. These results show that even "precise" numerical estimates of likelihood are not necessarily evaluable by consumers and that perceptions of likelihood are affected by supporting narrative information. Numeracy also moderated the effects of stated likelihood and the narrative evidence summary. Consumers higher in numeracy were more likely to use the stated likelihood information and consumers lower in numeracy were more likely to use the narrative evidence to inform their judgments. The moderating effect of likelihood format and consumer's perceptions of forecasts in hindsight are also explored. Explicit estimates of uncertainty are not necessarily useful to all intelligence consumers, particularly when presented with supporting narrative evidence. How consumers respond to intelligence forecasts depends on the structure of any supporting narrative information, the format of the explicit uncertainty information, and the numerical ability of the individual consumer. Forecasters should be sensitive to these three issues when presenting forecasts to consumers. / Adviser: Paul Slovic

Intelligence is vital for the outcome of battles. As long as humans wage war, there will be a need for decision support to military and civilian leaders regarding adversaries or potential adversaries. However, the production of intelligence is neither easy nor without pitfalls. There is a need to better understand the predicaments of intelligence analysis. Intelligence is bureaucratically produced as well as socially constructed and created in a distinct cultural context. The ‘institution’ captures these three aspects of influence. Therefore, with a particular focus on military intelligence, this thesis aims to deepen the understanding regarding institutional influence on intelligence assessments. The literature regarding intelligence has grown steadily over the last three decades. However, theories and frameworks aimed to understand the phenomenon are still sparse. This is even more true for literature regarding contemporary military intelligence. This thesis intends to contribute to bridging these research gaps. This is done by studying the Swedish military intelligence institution from several different perspectives: its rules-in-use, shared beliefs, and the incoming stimuli primarily related to conducting threat assessments. More precisely the thesis investigates the use of quantitative methods, doctrines (i.e. the formal rules), and shared beliefs connected to epistemological assumptions and threat assessments. The main contribution of this thesis is that it establishes and describes a casual link between a military intelligence institution and an assessment, by drawing upon rulesin-use and belief systems and their effect on the mental model and consequently the perception of the situation connected to a cognitive bias, and thereby its effect on a given assessment. The thesis makes an effort to render intelligence studies more generalizable, by way of adopting the Institutional Analysis and Development (IAD) framework. The metatheoretical language of the IAD is a promising avenue for explaining and describing the institutional influence on intelligence assessments. / Underrättelse är en avgörande komponent för utfallet av väpnad strid. Så länge människor krigar, kommer det att finnas ett behov av beslutsstöd till militära och civila ledare angående dess motståndare och potentiella motståndare. Produktionen av underrättelse är dock inte lätt eller utan fallgropar. Det finns där för ett behov av att öka förståelsen för de predikamenten kopplade till underrättelseanalys. Underrättelse som produkt är byråkratiskt såväl som socialt konstruerad och skapas i ett distinkt kulturellt sammanhang. Konceptet "Institution" kan ses fånga alla dessa tre aspekter. Därför handlar det speciellt om militär intelligens, som handlar om att förstå det institutionella inflytandet på intelligensbedömningar. Den tillgängliga underrättelselitteraturen har ökat stadigt under de senaste tre decennierna. Dock gällande teorier och ramverk på området som syftar till att förstå fenomenet är det emellertid fortfarande lite gjort. Detta gäller i än högre utsträckning för det specifika området modern militärunderrättelse verksamhet. Avhandlingen avser att bidra till att överbrygga dessa forskningsgap. Detta görs genom att studera den svenska militärunderrättelseinstitutionen ur flera perspektiv. Dess regler-i-bruk, delad trossystem/övertygelser samt den inkommande stimuli(data/information) primärt kopplade till hur hotbedömningar genomförs. Mer exakt granskar avhandlingen användningen av kvantitativa metoder, doktriner (dvs de formella reglerna) och delade föreställningar kopplade till epistemologiska antaganden och hotbedömningar. Huvudresultatet av denna avhandling är att det etablerar och beskriver en länk mellan en militärunderrättelseinstitution och de bedömningar som görs. Det går att se en direkt länk mellan de regler-i-bruk samt institutionens trossystem och deras inverkan på individens mentalmodellen. Detta sker genom att de rådande reglerna påverkar förekomesten av kognitivt bias vilket där med påverkar analytikerns uppfattning av en given situation. Avhandlingen har där med en ambition att göra studier i underrättelseanalys mer generaliserbara, genom att applicera och utveckla ramverket för institutionell analys och utveckling (IAD). Det metadeteoretiska språket i IAD är en lovande aveny för att förklara och beskriva det institutionella inflytandet på intelligensbedömningar.

The increasing volume and complexity of text data are challenging the cognitive capabilities of expert analysts. Machine learning and crowdsourcing present new opportunities for large-scale sensemaking, but it remains a challenge to model the overall process so that many distributed agents can contribute to suitable components asynchronously and meaningfully. In this work, I explore how to crowdsource sensemaking for intelligence analysis. Specifically, I focus on the complex processes that include developing hypotheses and theories from a raw dataset and iteratively refining the analysis. I first developed Connect the Dots, a web application that implements the concept of "context slices" and supports novice crowds in building relationship networks for exploratory analysis. Then I developed CrowdIA, a software platform that implements the entire crowd sensemaking pipeline and the context slicing for each step, to enable unsupervised crowd sensemaking. Using the pipeline as a testbed, I probed the errors and bottlenecks in crowdsourced sensemaking,and suggested design recommendations for integrated crowdsourcing systems. Building on these insights and to support iterative crowd sensemaking, I developed the concept of "crowd auditing" in which an auditor examines a pipeline of crowd analyses and diagnoses the problems to steer future refinement. I explored the design space to support crowd auditing and developed CrowdTrace, a crowd auditing tool that enables novice auditors to effectively identify the important problems with the crowd analysis and create microtasks for crowd workers to fix the problems.The core contributions of this work include a pipeline that enables distributed crowd collaboration to holistic sensemaking processes, two novel concepts of "context slices" and "crowd auditing", web applications that support crowd sensemaking and auditing, as well as design implications for crowd sensemaking systems. The hope is that the crowd sensemaking pipeline can serve to accelerate research on sensemaking, and contribute to helping people conduct in-depth investigations of large collections of information. / Doctor of Philosophy / In today's world, we have access to large amounts of data that provide opportunities to solve problems at unprecedented depths and scales. While machine learning offers powerful capabilities to support data analysis, to extract meaning from raw data is cognitively demanding and requires significant person-power. Crowdsourcing aggregates human intelligence, yet it remains a challenge for many distributed agents to collaborate asynchronously and meaningfully. The contribution of this work is to explore how to use crowdsourcing to make sense of the copious and complex data. I first implemented the concept of ``context slices'', which split up complex sensemaking tasks by context, to support meaningful division of work. I developed a web application, Connect the Dots, which generates relationship networks from text documents with crowdsourcing and context slices. Then I developed a crowd sensemaking pipeline based on the expert sensemaking process. I implemented the pipeline as a web platform, CrowdIA, which guides crowds to solve mysteries without expert intervention. Using the pipeline as a testbed, I probed the errors and bottlenecks in crowd sensemaking and provided design recommendations for crowd intelligence systems. Finally, I introduced the concept of ``crowd auditing'', in which an auditor examines a pipeline of crowd analyses and diagnoses the problems to steer a top-down path of the pipeline and refine the crowd analysis. The hope is that the crowd sensemaking pipeline can serve to accelerate research on sensemaking, and contribute to helping people conduct in-depth investigations of large collections of data.

The U.S. Intelligence Community is undergoing an "Analytic Transformation" designed to improve the quality of intelligence analysis. Information foraging theory, a human analogue to foraging theory that finds humans to be time- and risk-sensitive information seekers, is particularly relevant to this effort because it addresses two basic challenges that continually confront intelligence analysts: information overload and severe time constraints. The present investigation marks the first empirical foray into testing a theory of intelligence foraging. Two experiments using computer simulations tested the effects of temporal barriers on expert (intelligence analysts) and novice (undergraduates) search, consumption, and patch residence behaviors across three fictional databases (i.e., patch) containing information on the cause of a battleship explosion. The original hypotheses were not confirmed; handling time and travel time manipulations (in the form of different download delays associated with each database) did not significantly affect their database navigation patterns or their assessment of the battleship explosion. Unexpectedly, the specific content of each patch appeared to control their search and consumption behavior rather than the handling or travel time associated with each patch; the content effect mimicked the delay effect that was initially predicted. In the face of high stakes and realistic information constraints, the present study hints at an evolved information forager - one who is still content-driven in spite of severe time constraints. In light of the present findings and in service to our national security interests, future research would benefit from a deeper dive into information foraging situations with these new types of constraints. / Psychology

Information visualization is applied in many fields to gain faster insights with lighter user cognitive loads in analyzing large sets of data. As more products are being introduced each year, how can one select the most effective tool or representation form for the task? There are a number of information visualization evaluation methods currently available. However, these evaluation methods are often limited by the appropriateness of the tool for a given domain since they are not evaluating according to tools' intended use. Current methods conduct evaluations in a laboratory environment with "benchmark" tasks and often with field data sets not aligned with the intended use of the tools. The absence of realistic data sets and routine tests reduces the effectiveness of the evaluation in terms of the appropriateness of the tool for a given domain. Intended use evaluation approach captures the key activities that will use the visual technology to calibrate the evaluation criteria toward these first-order needs. This research thesis presents the results from an investigation into an intended use evaluation approach and its effectiveness of measuring domain specific information visualization tools. In investigating the evaluation approach, criteria for the intelligence analysis community have been developed for demonstration purposes. While the observations from this research are compelling for the intelligence community, the principles of the evaluation approach should apply to a wider range of visualization technologies. All the design rationale and processes were captured in this thesis. This thesis presents a design process of developing criteria and measuring five intelligence analysis visual analytic tools. The study suggests that in selecting and/or evaluating visual analytic tools, a little up front effort to analyze key activities regarding the domain field will be beneficial. Such analysis can substantially reduce evaluation time and necessary effort throughout a longer period of time. / Master of Science

The Swedish Military Intelligence and Security Directorate (MUST) is a producer of knowledge, a knowledge that is fundamental for decisionmaking in foreign and security policy. The intelligence knowledge production is often held as objective, value neutral, and with the intention of ‘speaking truth onto power’. However, this study holds that such a perspective on intelligence knowledge production calls for a revision. Hence, the overall purpose of this study is to examine the characteristics of knowledge in intelligence analysis and also to investigate how that knowledge is affected by the social context of its production, the military intelligence service. The source material is of three kinds; first texts and documents, second interviews with intelligence analysts and managers, and third observations of seminars and meetings during the production process of estimates. The results are that there is a strong presence of an implicit interpretive framework that continuously influences and guides the knowledge production and thereby makes the knowledge dependent on one specific perspective contrary to the intentional objectivity within the intelligence service. Further, the study reveals that the social and discursive practices for intelligence knowledge production include a ‘logic of appropriateness’ suggesting the presence of a structured Denkkollektiv with a structured Denkstil. The actions and choices of the individuals are transformed to create conformity to the norms within the social discursive practices. Thus, the inherited frame of interpretation, as well as the socialised norm of staying within the existing accepted frames ofthinking and acting ends up to the stability and duration of the not always accurate and fruitful Denkstil. At the core of political science resides the question of how policy is shaped. Even though this study has focused merely on one organisation in a specific policy field in one country it brings insights to the knowledge and policy nexus.

Thesis (MPhil (Information Science))--University of Stellenbosch, 2010. / ENGLISH ABSTRACT: The intelligence community throughout the world is still reeling after several intelligence failures. Proposals to improve Intelligence Analysis have had little impact as analysts, their managers and their organisations continue to cling to outdated threat perceptions, methodologies and organisational structures and cultures. This thesis looks through the lens of Knowledge Management at the various challenges that the Intelligence Analysis practice is faced with in the Knowledge Age. Firstly, theories and concepts from Intelligence Analysis are challenged when compared with those in Knowledge Management and the possibility of applying new vocabularies in intelligence is discussed. The second challenge intelligence analysts face is to understand and adapt to the changed world with its connected, non-linear and rapidly enfolding events and patterns which broadens their scope to a multi-faceted, complex and multi-disciplinary threat picture. The third challenge is to re-look the existing analytical methodologies, tools and techniques, realising that these are most probably inadequate in a complex environment. The fourth challenge Intelligence Analysis faces is to reach out to other disciplines and assess how new analytical techniques, both intuitive and structured, as well as cognitive models, collaborative and organisational structure concepts from within the Knowledge Management discipline can improve Intelligence Analysis’ grasp of the Knowledge Age. In conclusion, it is argued that intelligence analysts might be ready to reinvent themselves to address Knowledge Age issues, but that intelligence organisations are not able to support a new intelligence paradigm while still clinging to threat perceptions and structures befitting the Cold War. / AFRIKAANSE OPSOMMING: Die internasionale intelligensie gemeenskap steier steeds na verskeie intelligensie terugslae die afgelope dekade. Voorstelle om intelligensie analise te verbeter het weinig impak terwyl analiste, hulle bestuurders en organisasies voortgaan om vas te hou aan uitgediende bedreigingsperspesies, analitiese metodes en organisatoriese strukture en kulture. Deur die lens van Kennis Bestuur, poog hierdie verhandeling om die verskeie uitdagings wat die Intelligensie Analise praktyk in die Kennis Era in die gesig staar, te identifiseer. Eerstens word bestaande teorieë en konsepte in Intelligensie Analise met dié in Kennis Bestuur vergelyk en die moontlikheid van ‘n nuwe woordeskat vir intelligensie word bespreek. Die tweede uitdaging vir intelligensie analiste is om by die nuwe wêreld en versnellende verandering aan te pas. Hulle word nou gekonfronteer met ‘n bedreigingsprent wat veelvlakkig, kompleks en multi-dissiplinêr is. Die derde uitdaging is om die bestaande analitiese metodologiëe, hulpmiddels en tegnieke te herwaardeer in die lig van hierdie nuwe wêreld. Die vierde uitdaging is om na ander dissiplines, insluitend dié van Kennis Bestuur, uit te reik sodat Intelligensie Analise verbeter kan word deur die toepassing van hierdie dissiplines se analitiese metodes (beide intuitief en gestruktureerd), hul kognitiewe en samewerkings modelle, sowel as organisasie struktuur konsepte. Laastens word geargumenteer dat Intelligensie Analiste dalk gereed is om hulself te vernuwe, maar dat hul intelligensie organisasies nie ‘n nuwe intelligensie paradigma kan ondersteun terwyl hulle voortgaan om bedreigingspersepsies, strukture en bestuurbeginsels toe te pas wat eerder by die Koue Oorlog tuis hoort nie.