<p>Human multi-robot systems constitute a relatively new area of research that focuses on the interaction and collaboration between humans and multiple robots. Well-designed systems can enable a team of humans and robots to effectively work together on complex and sophisticated tasks such as exploration, monitoring, and search and rescue operations. This dissertation introduces an affective workload allocation system capable of adaptively allocating workload in real-time while considering the conditions and work performance of human operators in multi-human multi-robot teams. The proposed system is largely composed of three parts, taking the surveillance scenario involving multi-human operators and multi-robot system as an example. The first part of the system is a framework for an adaptive multi-human multi-robot system that allows real-time measurement and communication between heterogeneous sensors and multi-robot systems. The second part is an algorithm for real-time monitoring of humans' affective states using machine learning techniques and estimation of the affective state from multimodal data that consists of physiological and behavioral signals. The third part is a deep reinforcement learning-based workload allocation algorithm. For the first part of the affective workload allocation system, we developed a robot operating system (ROS)-based affective monitoring framework to enable communication among multiple wearable biosensors, behavioral monitoring devices, and multi-robot systems using the real-time operating system feature of ROS. We validated the sub-interfaces of the affective monitoring framework through connecting to a robot simulation and utilizing the framework to create a dataset. The dataset included various visual and physiological data categorized on the cognitive load level. The targeted cognitive load is stimulated by a closed-circuit television (CCTV) monitoring task on the surveillance scenario with multi-robot systems. Furthermore, we developed a deep learning-based affective prediction algorithm using the physiological and behavioral data captured from wearable biosensors and behavior-monitoring devices, in order to estimate the cognitive states for the second part of the system. For the third part of the affective workload allocation system, we developed a deep reinforcement learning-based workload allocation algorithm to allocate optimal workloads based on a human operator's performance. The algorithm was designed to take an operator's cognitive load, using objective and subjective measurements as inputs, and consider the operator's task performance model we developed using the empirical findings of the extensive user experiments, to allocate optimal workloads to human operators. We validated the proposed system through within-subjects study experiments on a generalized surveillance scenario involving multiple humans and multiple robots in a team. The multi-human multi-robot surveillance environment included an affective monitoring framework and an affective prediction algorithm to read sensor data and predict human cognitive load in real-time, respectively. We investigated optimal methods for affective workload allocations by comparing other allocation strategies used in the user experiments. As a result, we demonstrated the effectiveness and performance of the proposed system. Moreover, we found that the subjective and objective measurement of an operator's cognitive loads and the process of seeking consent for the workload transitions must be included in the workload allocation system to improve the team performance of the multi-human multi-robot teams.</p>
| Identifer | oai:union.ndltd.org:purdue.edu/oai:figshare.com:article/21688367 |
| Date | 17 May 2024 |
| Creators | Wonse Jo (13119627) |
| Source Sets | Purdue University |
| Detected Language | English |
| Type | Text, Thesis |
| Rights | CC BY 4.0 |
| Relation | https://figshare.com/articles/thesis/Affective_Workload_Allocation_System_For_Multi-human_Multi-robot_Teams/21688367 |
Page generated in 0.0016 seconds