Human-machine teaming aims to meld human cognitive strengths and the unique capabilities of smart machines to create intelligent teams adaptive to rapidly changing circumstances. One major contributor to the problem of human-machine teaming is a lack of communication skills on the part of the machine. The primary objective of this research is focused on a machine’s interruption timings or when a machine should share and communicate information with human teammates within human-machine teaming interactions. Previous work addresses interruption timings from the perspective of single human, multitasking and multiple human, single task interactions. The primary aim of this dissertation is to augment this area by approaching the same problem from the perspective of a multiple human, multitasking interaction. The proposed machine is the Collaborative Communication Interruption Management System (C-CIMS) which is tasked with leveraging speech information from a human-human task and making inferences on when to interrupt with information related to an orthogonal human-machine task. This study and previous literature both suggest monitoring task boundaries and engagement as candidate moments of interruptibility within multiple human, multitasking interactions. The goal then becomes designing an intermediate step between human teammate communication and points of interruptibility within these interactions. The proposed intermediate step is the mapping of low-level speech information such as prosodic and lexical information onto higher constructs indicative of interruptibility. C-CIMS is composed of a Task Boundary Prosody Model, a Task Boundary Topic Model, and finally a Task Engagement Topic Model. Each of these components are evaluated separately in terms of how they perform within two different simulated human-machine teaming scenarios and the speed vs. accuracy tradeoffs as well as other limitations of each module. Overall the Task Boundary Prosody Model is tractable within a real-time system because of the low-latency in processing prosodic information, but is less accurate at predicting task boundaries even within human-machine interactions with simple dialogue. Conversely, the Task Boundary and Task Engagement Topic Models do well inferring task boundaries and engagement respectively, but are intractable in a real-time system because of the bottleneck in producing automatic speech recognition transcriptions to make interruption decisions. The overall contribution of this work is a novel approach to predicting interruptibility within human-machine teams by modeling higher constructs indicative of interruptibility using low-level speech information.

Interruptions in time-critical, dynamic, and collaborative environments, such as air traffic control (ATC), can provide valuable, task-relevant information. However, they also negatively impact task performance by distracting the operator from on-going tasks and consuming attention resources. This thesis develops and assesses a tool to assist radar air traffic controllers in managing interruptions. Field observations and interviews with air traffic controllers were utilized to develop an understanding of how interruptions occur in real ATC environments, and to identify where opportunities exist to use technology to support the interruption management process. It was identified that operators in these environments could better manage the effects of interruptions if there were indications to one operator of the availability of a collaborator and the urgency of an interruption from a collaborator. Present communication systems do not facilitate the awareness of these functionalities. An initial prototype for providing these functionalities in operational ATC displays was designed. Feedback on the prototypes was solicited through Participatory Design (PD) sessions with air traffic controllers. Based on the refinement of these prototypes, the Working Awareness Interruption Tool (WAIT) was developed to support more efficient and appropriate interruption timing in the context of complex, real-time, distributed, human operator interactions. Variations of the tool demonstrated several ways of showing the availability of the controller to be interrupted (either through manual settings or automatic detection) as well as incorporating a means of conveying the urgency level of the interruption. In order to examine the utility of the tool and to assess the importance and validity of its features, an experiment was conducted in a laboratory-based setting. The results of the experiment show the potential of this tool in an environment representative of air traffic control tasks and communication. Although the sample size was limited, the WAIT facilitated improved performance on both objective measures and self-reported measures, and reduced the distraction effects of interruptions from other operators. These improvements occurred without affecting perceptions of the effectiveness of communications. Questionnaire and interview results showed that participants appear to prefer an automated setting of availability to be shown to other collaborators. Identifying two examples of key features supporting interruption management (communicating availability and urgency) in air traffic control is one of the key contributions of this work. The work also makes a contribution by demonstrating that providing a tool incorporating these features can improve performance in an environment representative of ATC, albeit with naïve participants. Finally, the research makes a contribution by presenting the challenges associated with evaluating interruption management tools that require collaboration between operators in a system.

Due to a current lack of principle-driven multimodal user interface design guidelines, designers may encounter difficulties when choosing the most appropriate display modality for given users or specific tasks (e.g., verbal versus spatial tasks). The development of multimodal display guidelines from both a user and task domain perspective is thus critical to the achievement of successful human-system interaction. Specifically, there is a need to determine how to design task information presentation (e.g., via which modalities) to capitalize on an individual operator's information processing capabilities and the inherent efficiencies associated with redundant sensory information, thereby alleviating information overload. The present effort addresses this issue by proposing a theoretical framework (Architecture for Multi-Modal Optimization, AMMO) from which multimodal display design guidelines and adaptive automation strategies may be derived. The foundation of the proposed framework is based on extending, at a functional working memory (WM) level, existing information processing theories and models with the latest findings in cognitive psychology, neuroscience, and other allied sciences. The utility of AMMO lies in its ability to provide designers with strategies for directing system design, as well as dynamic adaptation strategies (i.e., multimodal mitigation strategies) in support of real-time operations. In an effort to validate specific components of AMMO, a subset of AMMO-derived multimodal design guidelines was evaluated with a simulated weapons control system multitasking environment. The results of this study demonstrated significant performance improvements in user response time and accuracy when multimodal display cues were used (i.e., auditory and tactile, individually and in combination) to augment the visual display of information, thereby distributing human information processing resources across multiple sensory and WM resources. These results provide initial empirical support for validation of the overall AMMO model and a sub-set of the principle-driven multimodal design guidelines derived from it. The empirically-validated multimodal design guidelines may be applicable to a wide range of information-intensive computer-based multitasking environments.