Robots, including UAVs, have found increasing use in helping humans with dangerous and difficult tasks. The number of robots in use is increasing and is likely to continue increasing in the future. As the number of robots increases, human operators will need to coordinate and control the actions of large teams of robots. While multi-robot supervisory control has been widely studied, it requires that an operator divide his or her attention between robots. Consequently, the use of multi-robot supervisory control is limited by the number of robots that a human or team of humans can reasonably control. Swarm robotics -- large numbers of low-cost robots displaying collective behaviors -- offers an alternative approach by providing the operator with a small set of inputs and parameters that alter the behavior of a large number of autonomous or semi-autonomous robots. Researchers have asserted that this approach is more scalable and offers greater promise for managing huge numbers of robots. The emerging field of Human-Swarm Interaction (HSI) deals with the effective management of swarms by human operators. In this thesis we offer foundational work on the effect of HSI (a) on the individual robots, (b) on the group as a whole, and (c) on the workload of the human operator. We (1) show that existing general swarm algorithms are feasible on existing robots and can display collective behaviors as shown in simulations in the literature, (2) analyze the effect of interaction style and neighborhood type on the swarm's topology, (3) demonstrate that operator workload stays stable as the size of the swarm increases, but (4) find that operator workload is influenced by the interaction style. We also present considerations for swarm deployment on real robots.

Augmented Reality (AR) head-mounted display (HMD) provides users with an immersive virtual experience in the real world. The portability of this technology affords various information display options for construction workers that are not possible otherwise. The information delivered via an interactive user interface provides an innovative method to display complex building instructions, which is more intuitive and accessible compared with traditional paper documentations. However, there are still challenges hindering the practical usage of this technology at the construction jobsite. As a technical restriction, current AR HMD products have a limited field of view (FOV) compared to the human vision range. It leads to an uncertainty of how the obstructed view of display will affect construction workers' perception of hazards in their surrounding area. Similarly, the information displayed to workers requires rigorous testing and evaluation to make sure that it does not lead to information overload. Therefore, it is essential to comprehensively evaluate the impacts of using AR HMD from both perspectives of task performance and cognitive performance. This dissertation aims to bridge the gap in understanding the cognitive impacts of using AR HMD in construction assembly tasks. Specifically, it focuses on answering the following two questions: (1) How are task performance and cognitive skills affected by AR displays under complex working conditions? (2) How are moment-to-moment changes of mental workload captured and evaluated during construction assembly tasks? To answer these questions, this dissertation proposed two experiments. The first study tests two AR displays (conformal and tag-along) and paper instruction under complex working conditions, involving different framing scales and interference settings. Subjective responses are collected and analyzed to evaluate overall mental workload and situation awareness. The second study focuses on exploring an electroencephalogram (EEG) based approach for moment-to-moment capture and evaluation of mental workload. It uncovers the cognitive change on the time domain and provides room for further quantitative analyzing on mental workload. Especially, two frameworks of mental workload prediction are proposed by using (1) Long Short-Term Memory (LSTM) and (2) one-dimensional Convolutional Neural Network (1D CNN)-LSTM for forecasting EEG signal and, classifying task conditions and mental workload levels respectively. The approaches are tested to be effective and reliable for predicting and recognizing subjects' mental workload during assembly. In brief, this research contributes to the existing knowledge with an assessment of AR HMD use in construction assembly, including task performance evaluation and both subjective and physiological measurements for cognitive skills. / Doctor of Philosophy / Augmented Reality (AR) is an emerging technology that bridges the gap between virtual creatures and physical world with an immersive display experience. Today, head-mounted display (HMD) is well developed to meet the demands for portable AR devices. It provides interactive and intuitive display of 2D graphical information to make it easier to understand for users. Therefore, AR display has been studied in the past few years for a more simplified and productive construction assembly process. However, given the premise that construction is a high-risk industry, introducing such display technology to the jobsite needs to be carefully tested. One obstacle in current AR HMD products is the restriction of field of view (FOV), which may block users' view in presenting large-scale 3D objects. In construction assembly, workers need to deal with tasks in different scopes, such as wood framing for a residential house. Consequently, it is necessary to study how such technical challenge will impact workers' performance under different task conditions. Another concern comes from the mental perspective. Although AR display may bring convenience in acquiring effective information, it is difficult to measure if this generates excessive mental burden to users. Especially for construction workers, whether the overlaid display will cause distraction and information overload is crucial for protecting workers from hazards. To address the problems, this dissertation explores the gap in previous literature, where mental workload is not well studied for using AR HMD in construction assembly. Two experiments are conducted to comprehensively evaluate the impacts of AR displays on both assembly performance and users' mental status. The outcomes bring implications to theoretical and practical aspects. First, it compares two AR displays (2D tag-along image and 3D conformal model) with traditional paper documentation for assembly performance (efficiency and accuracy) and users' cognitive skills (mental workload and situation awareness). The findings revealed the impact of FOV restriction and provided a strategic solution to selecting display method for different task conditions. Second, it proposes a physiological approach to calculate mental workload from analyzing the features from brain waves. It uncovered the latent mental changes during the assembly. Furthermore, two deep learning approaches are applied to predict and classify mental workload. The prediction model depicted the trend of mental workload in eighteen seconds based on an eighty-four-second training set, while the classifier recognized two task conditions with different mental workload levels with an accuracy of 93.6%. The results have promising potential for future research in detecting and preventing abnormality in workers' mental status. In addition, it is generalizable to apply in other construction tasks and AR applications.

Assessing the pilot’s cognitive state is of increasing importance in aviation, especially for the development of adaptive assistance systems. For this purpose, the assessment of mental workload (MWL) is of special interest as an indication when and how to adapt the automation to fit the pilot’s current needs. Thus, there is a need to assess the pilot continuously, objectively and non-intrusively. Neurophysiological measurements like electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) are promising candidates for such an assessment. Yet, there is evidence that EEG- and fNIRS-based MWL measures are susceptible to influences from other concepts like mental fatigue (MF), and decrease in accuracy when MWL and MF confound. Still, there are only few studies targeting this problem, and no systematic investigation into this problem has taken place. Thus, the validity of neurophysiological MWL measures is not clear yet. In order to undertake such a systematic investigation, I conducted three studies: one experiment in which I investigated the effects of increasing MWL on cortical activation when MF is controlled for; a second experiment in which I examined the effects of increasing MF on cortical activation when MWL is controlled for; and a further comparative analysis of the gathered data. In order to induce MWL and MF in a controllable and comparable fashion, I conceived and used a simplified simulated flight task with an incorporated adapted n-back and monitoring task. I used a concurrent EEG-fNIRS measurement to gain neurophysiological data, and collected performance data and self-reported MWL and MF. In the first study (N = 35), I induce different four levels of MWL by increasing the difficulty of the n-back task, and controlled for MF by means of randomization and a short task duration (≤ 45 minutes). Higher task difficulty elicited higher subjective MWL ratings, declining performance, increased frontal theta band power and decreased frontal deoxyhaemoglobin (HbR) concentration. Furthermore, fNIRS proved more sensitive to tasks with low difficulty, and EEG to tasks with high difficulty. Only the combination of both methods was able to discriminate all four induced MWL levels. Thus, frontal theta band power and HbR were sensitive to changing MWL. In the second study (N = 31), I. I induced MF by means of time on task. Thus, I prolonged the task duration to approx. 90 minutes, and controlled for MWL by using a low but constant task difficulty derived from the first experiment. Over the course of the experiment, the participants’ subjective MF increased linearly, but their performance remained stable. In the EEG data, there was an early increase and levelling in parietal alpha band power and a slower, but steady increase in frontal theta band power. The fNIRS data did not show a consistent trend in any direction with increasing MF. Thus, only parietal alpha and frontal theta band power were sensitive to changing MF. In the third study, I investigated the validity of two EEG indices commonly used for MWL assessment, the Task Load Index (TLI) and the Engagement Index (EI). I computed the indices from the data of the two experiments, and compared the results between the datasets, and to single band powers. The TLI increased with increasing MWL, but was less sensitive than theta band power alone, and varied slightly with increasing MF. The EI did not vary with MWL, and was not sensitive to gradually increasing MF. Thus, neither index could be considered a valid MWL measure. In sum, neurophysiological measures can be used to assess changes in MWL. Yet, frontal HbR was the only measure sensitive to MWL that did not also vary with MF, and further research is needed to conclude if this finding holds true under different task characteristics. Thus, the tested EEG and fNIRS measures are only valid indications of MWL when confounding effects of MF are explicitly controlled for. I discuss further influences on the tested EEG and fNIRS measures, possible combinations with other data sources, and practical challenges for a neurophysiological MWL assessment. I conclude that neurophysiological measures should be used carefully outside the laboratory, as their validity will likely suffer in realistic settings. When their limitations are understood and respected, they can help to understand the cognitive processes involved in MWL, and can be a valuable addition to an MWL assessment.

Effective mental workload measurement is critical because mental workload significantly affects human performance. A non-invasive and objective workload measurement tool is needed to overcome limitations of current mental workload measures. Further, training/learning increases mental workload during skill or knowledge acquisition, followed by a decreased mental workload, though sufficient training times are unknown. The objectives of this study were to: (1) investigate the efficacy of using thermography as a non-contact physiological measure to quantify mental workload, (2) quantify and describe the relationship between mental workload and learning/training, and, (3) introduce a method to determine a sufficient training time and an optimal human performance level for a novel task by using thermography. Three studies were conducted to address these objectives. The first study investigated the efficacy of using thermography to quantity the relationship between mental workload and facial temperature changes while learning an alpha-numeric task. Thermography measured and quantified the mental workload level successfully. Strong and significant correlations were found among thermography, performance, and subjective workload measures (MCH and SWAT ratings). The second study investigated the utility of using a psychophysical approach to determine workload levels that maximize performance on a cognitive task. The second study consisted of an adjustment session (participants adjusted their own workload levels) and work session (participants worked at the chosen workload level). Participants were found to fall into two performance groups (low and high performers by accuracy rate) and results were significantly different. Thermography demonstrated whether both group found their optimal workload level. The last study investigated efficacy of using thermography to quantify mental workload level in a complex training/learning environment. Experienced drivers’ performance data was used as criteria to indicate whether novice drivers mastered the driving skills. Strong and significant correlations were found among thermography, subjective workload measures, and performance measures in novice drivers. This study verified that thermography is a reliable and valid way to measure workload as a non-invasive and objective method. Also, thermography provided more practical results than subjective workload measures for simple and complex cognitive tasks. Thermography showed the capability to identify a sufficient training time for simple or complex cognitive tasks.

No description available.

No description available.

No description available.

No description available.

No description available.

Modern complex safety-critical socio-technical systems (STSs) operate in an environment that requires high levels of human-machine interaction. Given the potential for catastrophic events , understanding human errors is a critical research area spanning disciplines such as management science, cognitive engineering, resilience engineering, and systems theory. However, a research gap remains when researching how errors impact system performance from a systemic perspective. This dissertation employs a systematic methodology and develops models that explore the relationship between errors and system performance, considering both macro-organizational and micro-worker perspectives. In Essay 1, the focus is on how firms respond to serious errors (catastrophic events), by exploring the oscillation behavior associated with the organizational learning and forgetting theory. The proposed simulation model contributes to the organizational science literature with a comprehensive approach that assesses the firm's response time to "serious" errors when the firm has a focus on safety with established safety thresholds. All of these considerations have subsequent impact on future performance. Essay 2 explores the relationship between safety-critical system's workers' workload, human error, and automation reliance for the Belgian railway traffic control center. Key findings include a positive relationship between traffic controller performance and workload, and an inverted U-shaped relationship with automation usage. This research offers new insights into the effects of cognitive workload and automation reliance in safety-critical STSs. Essay 3 introduces a calibrated System Dynamics model, informed by empirical data and existing theories on workload suboptimality. This essay contributes to the managerial understanding of workload management, particularly the feedback mechanism between operators' workload and human errors, which is driven by overload and underload thresholds. The model serves as a practical tool for managerial practitioners to estimate the likelihood of human errors based on workload distributions. Overall, this dissertation presents an interdisciplinary and pragmatic approach, blending theoretical and empirical methodologies. Its broad impacts extend across management science, cognitive engineering, and resilience engineering, contributing significantly to the understanding and management of safety-critical socio-technical systems. / Doctor of Philosophy / This dissertation is motivated by the increasing autonomy in infrastructure systems designed to enhance safety performance. Yet paradoxically, we continue to witness system failures leading to catastrophic disasters. High-profile incidents such as the Metro-North train derailment in New York City, the Boeing 737 MAX plane crashes, and the Challenger and Columbia space shuttle accidents highlight this contradiction. This research delves into safety-critical systems where the intricate collaboration between humans and machines is crucial, and where even minor human errors can lead to disastrous consequences. This dissertation is presented in three parts. In the first part I examine how firms react to serious errors. The study focuses on their learning processes following safety incidents and the potential for these lessons to be forgotten over time. I introduced a simulation model grounded in the organizational science literature, offering deeper insights into how companies respond to errors, including changes in safety focus, safety culture, and policy, and the impact of these factors on future company's performance. The second part shifts to a worker-centered perspective, exploring the relationship between workload, performance, and automation usage among traffic controllers. The findings indicate that while performance can improve with an increase in workload up to a certain threshold, excessive reliance on automation may lead to a decline in performance. This part of the study sheds light on how cognitive workload and technology usage influence operators in safety-critical roles. The final part of the dissertation presents another simulation model, this time focusing on how workload, and the resulting stress and boredom due to workload, influence the likelihood of errors. Utilizing real operational data from the Belgian railway transportation system, this model aids managers in understanding how to optimally balance workloads to minimize error risks. Overall, this dissertation takes an interdisciplinary and pragmatic approach, merging theoretical concepts with empirical data. Its extensive impact spans management science, cognitive engineering, and resilience engineering, significantly enhancing our comprehension and management of safety-critical socio-technical systems.