Mini Rotorcraft Unmanned Aerial Vehicles (MRUAVs) flown at low altitude as a part of a human-robot team are potential sources of tactical information for local search missions. Traditionally, their effectiveness in this role has been limited by an inability to intelligently perceive unknown environments or integrate with human team members. Human-robot collaboration provides the theory for building cooperative relationships in this context. This theory, however, only addresses those human-robot teams that are either robot-centered or human-centered in their decision making processes or relationships. This work establishes a new branch of human-robot collaborative theory, Operator Blending, which creates codependent and cooperative relationships between a single robot and human team member for tactical missions. Joint Intension Theory is the basis of this approach, which allows both the human and robot to contribute what each does well in accomplishing the mission objectives. Information processing methods for shared visual information and object tracking take advantage of the human role in the perception process. In addition, coupling of translational commands and the search process establish navigation as the shared basis of communication between the MRUAV and human, for system integration purposes. Observation models relevant to both human and robotic collaborators are tracked through a boundary based approach deemed AIM-SHIFT. A system is developed to classify the semantic and functional relevance of an observation model to local search called the Code of Observational Genetics (COG). These COGs are used to qualitatively map the environment through Qualitative Unsupervised Intelligent Collaborative Keypoint (QUICK) mapping, created to support these methods. / Ph. D.

Joint torque feedback (JTF) is a new and promising means of kinesthetic feedback to provide information to a person or guide them during a motion task. However, little work has been done to apply the torque feedback to a person. This project evaluates the properties of JTF as haptic feedback, starting from the fabrication of a lightweight elbow haptic exoskeleton. A cheap hobby motor and easily accessible hardware are introduced for manufacturing and open-sourced embedded architecture for data logging. The total cost and the weights are $500 and 509g. Also, as the prerequisite step to assess the JTF in guidance, human perceptual ability to detect JTF was quantified at the elbow during all possible static and dynamic joint statuses. JTF slopes per various joint conditions are derived using the Interweaving Staircase Method. For either directional torque feedback, flexional motion requires 1.89-2.27 times larger speed slope, in mNm/(°/s), than the extensional motion. In addition, we find that JTFs during the same directional muscle's isometric contraction yields a larger slope, in mNm/mNm, than the opposing direction (7.36 times and 1.02 times for extension torque and flexion torque). Finally, the guidance performance of the JTF was evaluated in terms of time delay and position error between the directed input and the wearer's arm. When studying how much the human arm travels with JTF, the absolute magnitude of the input shows more significance than the duration of the input (p-values of <0.0001 and 0.001). In the analysis of tracking the pulse input, the highest torque stiffness, 95 mNm/°, is responsible for the smallest position error, 6.102 ± 5.117°, despite the applied torque acting as compulsory stimuli. / Doctor of Philosophy / Joint torque feedback (JTF) is a new and promising means of haptic feedback to provide information to a person or guide them during a motion task. However, little work has been done to apply the torque feedback to a person, such as determining how well humans can detect external torques or how stiff the torque input should be to augment a human motion without interference with the voluntary movement. This project evaluates the properties of JTF as haptic feedback, starting from the fabrication of a lightweight elbow haptic exoskeleton. The novelty of the hardware is that we mask most of the skin receptors so that the joint receptors are primarily what the body will use to detect external sensations. A cheap hobby motor and easily accessible hardware are introduced for manufacturing and open-sourced software architecture for data logging. The total cost and the weight are $500 and 509g. Also, as the prerequisite step to assess the JTF in guidance, human perceptual ability to detect JTF was quantified at the elbow during all possible static and dynamic joint statuses. A psychophysics tool called Interweaving Staircase Method was implemented to derive torque slopes per various joint conditions. For either directional torque feedback, flexional motion requires 1.89-2.27 times larger speed slope, in mNm/(°/s) than the extensional motion. In addition, the muscles' isometric contraction with the aiding direction required a larger slope, in $mathrm{mNm/mNm}$ than the opposing direction (7.36 times and 1.02 times for extension torque and flexion torque). Finally, the guidance performance of the JTF was evaluated in terms of time delay and position error between the directed input and the wearer's arm. When studying how much the human arm travels with JTF, the absolute magnitude of the input shows more significance than the duration of the input (p-values of <0.0001 and 0.001). In the analysis of tracking the pulse input, the highest torque stiffness, 95 mNm/°, is responsible for the smallest position error, 6.102 ± 5.117°, despite the applied torque acting as compulsory stimuli.

The thesis focuses on applying deep learning and reinforcement learning techniques on human keypoint detection and robot arm manipulation. Inspired by Semi-Autonomous Victim Extraction Robot (SAVER), an autonomous first-aid airway-management robotic system designed to perform Cricothyrotomy on patients is proposed. Perception, decision-making, and control are embedded in the system. In this system, first, the location of the cricothyroid membrane (CTM)-the incision site of Cricothyrotomy- is detected; then, the robot arm is controlled to reach the detected position on a medical manikin. A hybrid neural network (HNNet) that can balance both speed and accuracy is proposed. HNNet is an ensemble-based network architecture that consists of two ensembles: the region proposal ensemble and the keypoint detection ensemble. This architecture can maintain the original high resolution of the input image without heavy computation and can meet the high-precision and real-time requirements at the same time. A dataset containing more than 16,000 images from 13 people, with a clear view of the neck area, and with CTM position labeled by a medical expert was built to train and validate the proposed model. It achieved a success rate of $99.6%$ to detect the position of the CTM with an error of less than 5mm. The robot arm manipulator was trained with the reinforcement learning model to reach the detected location. Finally, the detection neural network and the manipulation process are combined as an integrated system. The system was validated in real-life experiments on a human-sized medical manikin using a Kinect V2 camera and a MICO robot arm manipulator. / Master of Science / The thesis focuses on applying deep learning and reinforcement learning techniques on human keypoint detection and robot arm manipulation. Inspired by Semi-Autonomous Victim Extraction Robot (SAVER), an autonomous first-aid airway-management robotic system designed to perform Cricothyrotomy on patients is proposed. Perception, decision-making, and control are embedded in the system. In this system, first, the location of the cricothyroid membrane(CTM)-the incision site of Cricothyrotomy- is detected; then, the robot arm is controlled to reach the detected position on a medical manikin. A hybrid neural network (HNNet) that can balance both speed and accuracy is proposed. HNNet is an ensemble-based network architecture that consists of two ensembles: the region proposal ensemble and the keypoint detection ensemble. This architecture can maintain the original high resolution of the input image without heavy computation and can meet the high-precision and real-time requirements at the same time. Finally, the detection neural network and the manipulation process are combined as an integrated system. The robot arm manipulator was trained with the reinforcement learning model to reach the detected location. The system was validated in real-life experiments on a human-sized medical manikin using an RGB-D camera and a robot arm manipulator.

This thesis presents a framework for a mobile robot to escort a human to their destination successfully and efficiently. The proposed technique uses accurate path prediction incorporating human intention to locate the robot in front of the human while walking. Human intention is inferred by the head pose, an effective past-proven implicit indicator of intention, and fused with conventional physics-based motion prediction. The human trajectory is estimated and predicted using a particle filter because of the human's nonlinear and non-Gaussian behavior, and the robot control action is determined from the predicted human pose allowing for anticipative autonomous escorting. Experimental analysis shows that the incorporation of the proposed human intention model reduces human position prediction error by approximately 35% when turning. Furthermore, experimental validation with an omnidirectional mobile robotic platform shows escorting up to 50% more accurate compared to the conventional techniques, while achieving 97% success rate. / Master of Science / This thesis presents a method for a mobile robot to escort a human to their destination successfully and efficiently. The proposed technique uses human intention to predict the walk path allowing the robot to be in front of the human while walking. Human intention is inferred by the head direction, an effective past-proven indicator of intention, and is combined with conventional motion prediction. The robot motion is then determined from the predicted human position allowing for anticipative autonomous escorting. Experimental analysis shows that the incorporation of the proposed human intention reduces human position prediction error by approximately 35% when turning. Furthermore, experimental validation with an mobile robotic platform shows escorting up to 50% more accurate compared to the conventional techniques, while achieving 97% success rate. The unique escorting interaction method proposed has applications such as touch-less shopping cart robots, exercise companions, collaborative rescue robots, and sanitary transportation for hospitals.

The growing presence of modern learning robots necessitates a fundamental shift in design, as these robots must learn skills from human inputs. Two main components close the loop in a human-robot interaction: learning and communication. Learning derives robot behaviors from human inputs, and communication conveys information about the robot's learning to the human. This dissertation focuses on methods that enable robots to communicate their internal state clearly while learning precisely from human inputs. We first consider the information implicitly communicated by robot behavior during human interactions and whether it can be utilized to form human-robot teams. We investigate behavioral economics to identify biases and expectations in human team dynamics and incorporate them into human-robot teams. We develop and demonstrate an optimization approach that relates high-level subtask allocations to low-level robot actions, which implicitly communicates learning to encourage human participation in robot teams. We then study how communication helps humans teach tasks to robots using active learning and interactive imitation learning algorithms. Within the active learning approach, we develop a model that forms a belief over the human's mental model about the robot's learning. We validate that our algorithm enables the robot to balance between learning human preferences and implicitly communicating its learning through questions. Within the imitation learning approach, we integrate a wrapped haptic display that explicitly communicates representations from the robot's learned behavior to the user. We show that our framework helps the human teacher improve different aspects of the robot's learning during kinesthetic teaching. We then extend this system to a more comprehensive interactive learning architecture that provides multi-modal feedback through augmented reality and haptic interfaces. We present a case study with this closed-loop system and illustrate improved teaching, trust, and co-adaptation as the measured benefits of communicating robot learning. Overall, this dissertation demonstrates that bi-directional communication helps robots learn faster and adapt better, while humans experience a more intuitive and trust-based interaction. / Doctor of Philosophy / The growing presence of modern learning robots necessitates a fundamental shift in design, as these robots must learn skills from human inputs. This dissertation focuses on methods that enable robots to communicate their internal state clearly while learning precisely from human inputs. We first consider how robot behaviors during human interactions can be used to form human-robot teams. We investigate human-human teams in behavioral economics to better understand human expectations in human-robot teams. We develop a model that enables robots to distribute subtasks in a way that encourages their human partner to keep collaborating with them. We then study how communication helps human-in-the-loop robot teaching. Within active learning, we develop a model that infers what the human thinks about the robot's learning. We validate that, with our algorithm, the robot efficiently learns human preferences and keeps the human updated about what it has learned. Within imitation learning, we integrate a haptic device that explicitly communicates features from the robot's learned behavior to the user. We show that our framework helps users effectively improve their kinesthetic teaching. We then extend this system to a more comprehensive interactive robot learning architecture that provides feedback through augmented reality and haptic interfaces. We conduct a case study and illustrate that our framework improves robot teaching, human trust, and human-robot co-adaptation. Overall, this dissertation demonstrates that bi-directional communication helps robots learn faster and adapt better, while humans experience a more intuitive and trust-based interaction.

Autonomous robots have proven to be useful for search and rescue (SAR) by being deployed in emergency situations and removing the need for direct human presence. However, there remains a need for effective communication between the robot and human operator to cooperate as a team. This thesis investigates the question of when to explain an autonomous agent’s behaviour in the setting of human-robot teaming. A game environment is developed to conduct a virtual SAR experiment to test the effect of explanation timings between two conditions: (1) always explaining - the robot provides explanations whenever possible, and (2) anticipatory explaining - the robot determines when to explain based on the context. When to explain is determined through the construction of argumentation frameworks modelling the context. The effect of anticipatory information pushing is tested on two metrics: team performance and explanation experience. Results indicate anticipatory explaining does not have a significant effect on team performance and participants’ explanation satisfaction. Additionally, participant feedback shows they prefer to be in control instead of cooperating as a team. These findings underline the importance of studying explanation presentation in high-demanding environments and indicate a need for interdisciplinary discussion on the design of human-robot teaming.

A collaborative robot is something of growing interest for companies in the manufacturing industries to implement. However, a collaborative robot is quite new in today’s market. An issue that arises is that no implementation process for collaborative robots exists today, as well as no requirement guide for skills, as well as actors, has been defined. The aim of this project was to examine how an implementation process of collaborative robots in manufacturing companies could look like. Focusing on charting the integration process steps of a collaborative robot, and identifying the actors as well as skills needed for successful cobot integration, with the aim achieve the goal of this thesis by answering the research questions. The thesis had the following research questions: Research Question 1 – How is an integration process for implementing a cobot represented in the manufacturing companies? Research Question 2 – What particular skills as well as actors are required when implementing in a cobot in the manufacturing companies? To answer the research questions, the author conducted several interviews with different companies. The interview questions were mainly constructed in order to answer the RQs but also to get an understanding for the different aspects of what a cobot is, what is required as well as how it compares to a traditional industrial robot. The thesis resulted in an implementation process with several steps constructed in order to implement a cobot as well as different aspects of what skills and actors are needed. In order to separate the aspects, the respondents were categorized into different roles which are the developer, integrator and the user. The different roles were all vital, providing an understanding from different perspectives. Keywords: Collaborative robot, cobot, Human-robot interaction, Human-Robot Collaboration, Development strategies, Automation, Industry 4.0.

In der vorliegenden Dissertation wurde ein datengetriebenes Verfahren zum maschinellen Lernen von Mensch-Roboter Interaktionen auf Basis von Mensch-Mensch Demonstrationen entwickelt. Während einer Trainingsphase werden Bewegungen zweier Interakteure mittels Motion Capture erfasst und in einem Zwei-Personen Interaktionsmodell gelernt. Zur Laufzeit wird das Modell sowohl zur Erkennung von Bewegungen des menschlichen Interaktionspartners als auch zur Generierung angepasster Roboterbewegungen eingesetzt. Die Leistungsfähigkeit des Ansatzes wird in drei komplexen Anwendungen evaluiert, die jeweils kontinuierliche Bewegungskoordination zwischen Mensch und Roboter erfordern. Das Ergebnis der Dissertation ist ein Lernverfahren, das intuitive, zielgerichtete und sichere Kollaboration mit Robotern ermöglicht.

In zahlreichen Forschungsprojekten wird unter Einsatz großer finanzieller und personeller Ressourcen daran gearbeitet, dass Roboter die Fabrikhallen verlassen und Teil von Alltagswelten wie Krankenhäusern, Kindergärten und Privatwohnungen werden. Die Konstrukteurinnen und Konstrukteure stehen dabei vor einer nicht-trivialen Herausforderung: Sie müssen die Ambivalenzen und Kontingenzen alltäglicher Interaktion in die diskrete Sprache der Maschinen übersetzen. Wie sie dieser Herausforderung begegnen, welche Muster und Lösungen sie heranziehen und welche Implikationen für die Verwendung von Sozialrobotern dabei gelegt werden, ist der Gegenstand des Buches. Auf der Suche nach der Antwort, was Roboter sozial macht, hat Andreas Bischof Forschungslabore und Konferenzen in Europa und Nordamerika besucht und ethnografisch erforscht. Zu den wesentlichen Ergebnissen dieser Studie gehört die Typologisierung von Forschungszielen in der Sozialrobotik, eine epistemische Genealogie der Idee des Roboters in Alltagswelten, die Rekonstruktion der Bezüge zu 'echten' Alltagswelten in der Sozialrobotik-Entwicklung und die Analyse dreier Gattungen epistemischer Praktiken, derer sich die Ingenieurinnen und Ingenieure bedienen, um Roboter sozial zu machen.:EINLEITUNG 1. WAS IST SOZIALROBOTIK? 1.1 Roboter & Robotik zum Funktionieren bringen 1.2 Drei Problemdimensionen der Sozialrobotik 1.3 Forschungsstand Sozialrobotik 1.4 Problemstellung – Sozialrobotik als „wicked problem“ 2. FORSCHEN, TECHNISIEREN UND ENTWERFEN 2.1 Wissenschaft als (soziale) Praxis 2.2 Technisierung und Komplexitätsreduktion in Technik 2.3 Entwurf, Technik, Nutzung – Technik zwischen Herstellungs- und Wirkungszusammenhang 2.4 Sozialrobotik als Problemlösungshandeln 3. METHODOLOGIE UND METHODEN DER STUDIE 3.1 Forschungsstil Grounded Theory 3.2 Ethnografie und narrative Experteninterviews 3.3 Auswertungsmethoden und Generalisierung 3.4 Zusammenfassung 4. DER ROBOTER ALS UNIVERSALWERKZEUG 4.1 Roboter als fiktionale Apparate 4.2 Robotik als Lösungsversprechen 4.3 Computer Science zwischen Wissenschaft und Design 4.4 Fazit – Das Erbe des Universalwerkzeugs 5. FORSCHUNGS- UND ENTWICKLUNGSZIELE DER SOZIALROBOTIK 5.1 Bedingungen projektförmiger Forschung 5.2 Dimensionen und Typen der Ziele von Sozialrobotik 5.3 Beschreibung der Typen anhand der Verteilung der Fälle 5.4 Ko-Konstruktion der Anwendung an Fallbeispielen 5.5 Fazit – Typen von Sozialität in Entwicklungszielen 6. EPISTEMISCHE PRAKTIKEN UND INSTRUMENTE DER SOZIALROBOTIK 6.1 Praktiken der Laboratisierung des Sozialen 6.2 Alltägliche und implizite Heuristiken 6.3 Inszenierende Praktiken 6.4 Fazit – Wechselspiele des Erzeugens und Beobachtens 7. FAZIT 7.1 Phänomenstruktur der Sozialrobotik 7.2 Entwicklung als Komplexitätspendel 7.3 Methodologischer Vorschlag für den Entwicklungsprozess

L'interaction homme-robot arrive dans une phase intéressante ou la relation entre un homme et un robot est envisage comme 'un partenariat plutôt que comme une simple relation maitre-esclave. Pour que cela devienne une réalité, le robot a besoin de comprendre le comportement humain. Il ne lui suffit pas de réagir de manière appropriée, il lui faut également être socialement proactif. Pour que ce comportement puis être mise en pratique le roboticien doit s'inspirer de la littérature déjà riche en sciences sociocognitives chez l'homme. Dans ce travail, nous allons identifier les éléments clés d'une telle interaction dans le contexte d'une tâche commune, avec un accent particulier sur la façon dont l'homme doit collaborer pour réaliser avec succès une action commune. Nous allons montrer l'application de ces éléments au cas un système robotique afin d'enrichir les interactions sociales homme-robot pour la prise de décision. A cet égard, une contribution a la gestion du but de haut niveau de robot et le comportement proactif est montre. La description d'un modèle décisionnel d'collaboration pour une tâche collaboratif avec l'humain est donnée. Ainsi, l'étude de l'interaction homme robot montre l'intéret de bien choisir le moment d'une action de communication lors des activités conjointes avec l'humain / Human Robot Interaction is entering into the interesting phase where the relationship with a robot is envisioned more as one of companionship with the human partner than a mere master-slave relationship. For this to become a reality, the robot needs to understand human behavior and not only react appropriately but also be socially proactive. A Companion Robot will also need to collaborate with the human in his daily life and will require a reasoning mechanism to manage thecollaboration and also handle the uncertainty in the human intention to engage and collaborate. In this work, we will identify key elements of such interaction in the context of a collaborative activity, with special focus on how humans successfully collaborate to achieve a joint action. We will show application of these elements in a robotic system to enrich its social human robot interaction aspect of decision making. In this respect, we provide a contribution to managing robot high-level goals and proactive behavior and a description of a coactivity decision model for collaborative human robot task. Also, a HRI user study demonstrates the importance of timing a verbal communication in a proactive human robot joint action