<p>Human multi-robot systems constitute a relatively new area of research that focuses on the interaction and collaboration between humans and multiple robots. Well-designed systems can enable a team of humans and robots to effectively work together on complex and sophisticated tasks such as exploration, monitoring, and search and rescue operations. This dissertation introduces an affective workload allocation system capable of adaptively allocating workload in real-time while considering the conditions and work performance of human operators in multi-human multi-robot teams. The proposed system is largely composed of three parts, taking the surveillance scenario involving multi-human operators and multi-robot system as an example. The first part of the system is a framework for an adaptive multi-human multi-robot system that allows real-time measurement and communication between heterogeneous sensors and multi-robot systems. The second part is an algorithm for real-time monitoring of humans' affective states using machine learning techniques and estimation of the affective state from multimodal data that consists of physiological and behavioral signals. The third part is a deep reinforcement learning-based workload allocation algorithm. For the first part of the affective workload allocation system, we developed a robot operating system (ROS)-based affective monitoring framework to enable communication among multiple wearable biosensors, behavioral monitoring devices, and multi-robot systems using the real-time operating system feature of ROS. We validated the sub-interfaces of the affective monitoring framework through connecting to a robot simulation and utilizing the framework to create a dataset. The dataset included various visual and physiological data categorized on the cognitive load level. The targeted cognitive load is stimulated by a closed-circuit television (CCTV) monitoring task on the surveillance scenario with multi-robot systems. Furthermore, we developed a deep learning-based affective prediction algorithm using the physiological and behavioral data captured from wearable biosensors and behavior-monitoring devices, in order to estimate the cognitive states for the second part of the system. For the third part of the affective workload allocation system, we developed a deep reinforcement learning-based workload allocation algorithm to allocate optimal workloads based on a human operator's performance. The algorithm was designed to take an operator's cognitive load, using objective and subjective measurements as inputs, and consider the operator's task performance model we developed using the empirical findings of the extensive user experiments, to allocate optimal workloads to human operators. We validated the proposed system through within-subjects study experiments on a generalized surveillance scenario involving multiple humans and multiple robots in a team. The multi-human multi-robot surveillance environment included an affective monitoring framework and an affective prediction algorithm to read sensor data and predict human cognitive load in real-time, respectively. We investigated optimal methods for affective workload allocations by comparing other allocation strategies used in the user experiments. As a result, we demonstrated the effectiveness and performance of the proposed system. Moreover, we found that the subjective and objective measurement of an operator's cognitive loads and the process of seeking consent for the workload transitions must be included in the workload allocation system to improve the team performance of the multi-human multi-robot teams.</p>

In everyday communication, humans utilise eye gaze due to its importance as a communication tool. As technology evolves, social robots are expected to become more adopted in society and, since they interact with humans, they should similarly use eye gaze to elevate the level of the interaction and increase humans’ perception of them. Previous studies have shown that robots possessing human-like gaze behaviour increase the interactants’ task performance and their perception of the robot. However, social robots must also be able to behave and respond appropriately when humans act inappropriately, and failure in doing so may normalize bad behaviour even towards other humans. Additionally, with the recent progress of wearable eyetracking technology, there is interest to see how this technology can be used to help generate human gaze into a robot. This thesis work investigates how the eye gaze behaviour from a human being can be modeled into the robot Furhat to behave more human-like in a confrontational scenario. It further investigates how a robot possessing the developed human-like gaze model compares to a robot using a believable heuristic gaze model. We created a pipeline which concerned selecting scenarios, conducting roleplays between actors of these scenarios to collect gaze, extracting and processing that gaze data and extracting probability distributions that the human-like model would utilise. Our model used frequencies to determine where to gaze and head rotation, while gamma distributions were used to sample gaze length. We then executed an online video study with the two robot conditions where participants rated either robot by filling out a questionnaire. The results show that while no statistical difference could be found, the human-like condition scored higher on the measures anthropomorphism/human-likeness and composure, whereas the heuristic condition rated higher on expertise and extroversion. As such, the human-like model did not yield a significant benefit on robot perception to opt for it. Still, we suggest that the pipeline used in this thesis work may help HRI researchers to perform gaze studies and possibly build a foundation for further development. / I vardaglig kommunikation använder människor sig av blickar på grund av dess betydelse som kommunikationsverktyg. Då teknologi ständigt utvecklas förväntas det att sociala robotar kommer att bli mer involverade i samhället, och eftersom de integrerar med människor så bör de på samma sätt använda sig av blickar och ögonrörelser för att höja nivån på interaktionen och därmed förbättra människors uppfattning av dem. Tidigare studier har visat att robotar som använder sig av blickar likt människor kan förbättra deltagarnas utförande av uppgifter samt deras uppfattning av roboten. Sociala robotar måste dock även kunna agera och svara på ett lämpligt sätt när människor beter sig olämpligt, och gör dem inte det finns risken att det olämpliga beteendet normaliseras, även i interaktioner med andra människor. Med de senaste framstegen av portabla eye-tracking enheter finns det därför ett intresse att se hur denna teknologi kan användas för att generera människolikt blickbeteende som sedan används i en robot. Denna studie undersöker hur en människas sätt att blicka och titta kan modelleras i roboten Furhat för att bete sig mer människolikt i ett scenario där konfrontation behövs. Studien undersöker dessutom hur en robot som bär ett människolikt blickbeteende jämför sig med en robot som bär ett trovärdigt heuristiskt blickbeteende. Vi skapade en struktur som involverade att välja scenarion, utföra rollspel mellan skådespelare i dessa scenarier för att samla data om deras blickmönster, extrahera och bearbeta denna data, och extraherade sannolikhetsfördelningar som den människolika modellen skulle använda sig av. Vår modell använde sig av frekvenser för att besluta var roboten skulle blicka, medan gammafördelningar användes för att generera blickens längd. Vi utförde därefter en videostudie online med de två robotvarianterna, där deltagare bedömde någon av robotarna genom att svara på en enkät. Resultaten visar att ingen statistisk signifikant skillnad kunde påvisas. Trender visade dock att modellen med människolik blickbeteende bedömdes högre i mätningen av attributerna antropomorfism/mänsklighet och fattning, medan den heuristiska modellen bedömdes högre i expertis och utåtvändighet. Därav erhöll den människolika modellen ingen signifikant framgång för att föredra den. Vi föreslår ändå att strukturen som användes i studien kan hjälpa MRI forskare att utföra studier som involverar blickbeteende hos människor, och möjligtvis bygga en grund för vidareutveckling av strukturen.

This study investigates if preferences for mental capacities in robots vary across different application domains and identifies influential factors, both in individuals’ characteristics and attributes specific to each robot domain. Employing a between-subject design, participants (N=271) completed a survey collecting both quantitative and qualitative data on preferences for 12 mental capacities across six robot types situated in a specific application domain (medicine, defense, household, social, education, customer service). Half of the mental capacities align with each dimension (experience and agency) in the two-dimensional model of mind (Gray et al., 2007; McMurtrie, 2023). Key findings reveal a general preference for high agency ability and low experience ability across all application domains. Exceptions were found in preference for lower agency ability in the cleaning robot and higher experience ability in the companion robot. Qualitative analysis indicates a desire for objective and logical robots functioning without emotions, while demonstrating empathy for human emotions. Additionally, gender and educational background emerged as factors influencing preference for lower experience abilities in robots. While previous research has mainly focused on attribution of mental capacities to technical agents, this study provides insights into human preferences and factors affecting them. These insights can guide responsible and ethics-driven development and design of robot technology within the field of human-robot interaction.

Bientôt, les robots ne travailleront plus de manière isolée mais avec nous. Ils entrent peu à peu dans notre vie de tous les jours pour coopérer, assister, aider, servir, apprendre, enseigner ou même jouer avec l'homme. Dans ce contexte, nous considérons que ce ne doit pas être à l'homme de s'adapter au robot. Au contraire, le robot doit être capable d'intégrer, dans ses stratégies de planification et de décision, différents facteurs d'effort et de confort et de prendre en compte les préférences et désirs de l'homme ainsi que les normes sociales de son environnement. Tout en respectant les principes de sécurité réglementaire, le robot doit se comporter, naviguer, manipuler, communiquer et apprendre d'une manière qui soit pertinente, acceptée et compréhensible par l'homme. Cette thèse explore et définit les ingrédients clés nécessaires au robot pour développer une telle intelligence socio-cognitive. Elle définit également un cadre pour l'interaction homme-robot permettant de s'attaquer à ces challenges dans le but de rendre le robot socialement intelligent / Robots will no longer be working isolated from us. They are entering into our day-to-day life to cooperate, assist, help, serve, learn, teach and play with us. In this context, it is important that because of the presence of robots, the human should not be on compromising side. To achieve this, beyond the basic safety requirements, robots should take into account various factors ranging from human’s effort, comfort, preferences, desire, to social norms, in their various planning and decision making strategies. They should behave, navigate, manipulate, interact and learn in a way, which is expected, accepted, and understandable by us, the human. This thesis begins by exploring and identifying the basic yet key ingredients of such socio-cognitive intelligence. Then we develop generic frameworks and concepts from HRI perspective to address these additional challenges, and to elevate the robots capabilities towards being socially intelligent

The fast technology development for military unmanned ground vehicles (UGVs) has led to a considerable demand to explore the soldier’s role in an interactive UGV system. This thesis explores how to design interactive systems for UGVs for infantry soldiers in the Swedish Armed Force. This was done through a user-centered design approach in three steps; (1) identifying the design drivers of the targeted military context through qualitative observations and user interviews, (2) using the design drivers to investigate concepts for controlling the UGV, and (3) create and evaluate a prototype of an interactive UGV system design. Results from interviews indicated that design drivers depend on the physical and psychological context of the intended soldiers. In addition, exploring the different concepts showed that early conceptual designs helped the user express their needs of a non-existing system. Furthermore, the results indicate that an interactive UGV system does not necessarily need to be at the highest level of autonomy in order to be useful for the soldiers on the field. The final prototype of an interactive UGV system was evaluated using a demonstration video, a Technology Acceptance Model (TAM), and semi-structured user interviews. Results from this evaluation suggested that the soldiers see the potential usefulness of an interactive UGV system but are not entirely convinced. In conclusion, this thesis argues that in order to design an interactive UGV system, the most critical aspect is the soldiers’ acceptance of the new system. Moreover, for soldiers to accept the concept of military UGVs, it is necessary to understand the context of use and the needs of the soldiers. This is done by involving the soldiers already in the conceptual design process and then throughout the development phases.