Mini Rotorcraft Unmanned Aerial Vehicles (MRUAVs) flown at low altitude as a part of a human-robot team are potential sources of tactical information for local search missions. Traditionally, their effectiveness in this role has been limited by an inability to intelligently perceive unknown environments or integrate with human team members. Human-robot collaboration provides the theory for building cooperative relationships in this context. This theory, however, only addresses those human-robot teams that are either robot-centered or human-centered in their decision making processes or relationships. This work establishes a new branch of human-robot collaborative theory, Operator Blending, which creates codependent and cooperative relationships between a single robot and human team member for tactical missions. Joint Intension Theory is the basis of this approach, which allows both the human and robot to contribute what each does well in accomplishing the mission objectives. Information processing methods for shared visual information and object tracking take advantage of the human role in the perception process. In addition, coupling of translational commands and the search process establish navigation as the shared basis of communication between the MRUAV and human, for system integration purposes. Observation models relevant to both human and robotic collaborators are tracked through a boundary based approach deemed AIM-SHIFT. A system is developed to classify the semantic and functional relevance of an observation model to local search called the Code of Observational Genetics (COG). These COGs are used to qualitatively map the environment through Qualitative Unsupervised Intelligent Collaborative Keypoint (QUICK) mapping, created to support these methods. / Ph. D.

Joint torque feedback (JTF) is a new and promising means of kinesthetic feedback to provide information to a person or guide them during a motion task. However, little work has been done to apply the torque feedback to a person. This project evaluates the properties of JTF as haptic feedback, starting from the fabrication of a lightweight elbow haptic exoskeleton. A cheap hobby motor and easily accessible hardware are introduced for manufacturing and open-sourced embedded architecture for data logging. The total cost and the weights are $500 and 509g. Also, as the prerequisite step to assess the JTF in guidance, human perceptual ability to detect JTF was quantified at the elbow during all possible static and dynamic joint statuses. JTF slopes per various joint conditions are derived using the Interweaving Staircase Method. For either directional torque feedback, flexional motion requires 1.89-2.27 times larger speed slope, in mNm/(°/s), than the extensional motion. In addition, we find that JTFs during the same directional muscle's isometric contraction yields a larger slope, in mNm/mNm, than the opposing direction (7.36 times and 1.02 times for extension torque and flexion torque). Finally, the guidance performance of the JTF was evaluated in terms of time delay and position error between the directed input and the wearer's arm. When studying how much the human arm travels with JTF, the absolute magnitude of the input shows more significance than the duration of the input (p-values of <0.0001 and 0.001). In the analysis of tracking the pulse input, the highest torque stiffness, 95 mNm/°, is responsible for the smallest position error, 6.102 ± 5.117°, despite the applied torque acting as compulsory stimuli. / Doctor of Philosophy / Joint torque feedback (JTF) is a new and promising means of haptic feedback to provide information to a person or guide them during a motion task. However, little work has been done to apply the torque feedback to a person, such as determining how well humans can detect external torques or how stiff the torque input should be to augment a human motion without interference with the voluntary movement. This project evaluates the properties of JTF as haptic feedback, starting from the fabrication of a lightweight elbow haptic exoskeleton. The novelty of the hardware is that we mask most of the skin receptors so that the joint receptors are primarily what the body will use to detect external sensations. A cheap hobby motor and easily accessible hardware are introduced for manufacturing and open-sourced software architecture for data logging. The total cost and the weight are $500 and 509g. Also, as the prerequisite step to assess the JTF in guidance, human perceptual ability to detect JTF was quantified at the elbow during all possible static and dynamic joint statuses. A psychophysics tool called Interweaving Staircase Method was implemented to derive torque slopes per various joint conditions. For either directional torque feedback, flexional motion requires 1.89-2.27 times larger speed slope, in mNm/(°/s) than the extensional motion. In addition, the muscles' isometric contraction with the aiding direction required a larger slope, in $mathrm{mNm/mNm}$ than the opposing direction (7.36 times and 1.02 times for extension torque and flexion torque). Finally, the guidance performance of the JTF was evaluated in terms of time delay and position error between the directed input and the wearer's arm. When studying how much the human arm travels with JTF, the absolute magnitude of the input shows more significance than the duration of the input (p-values of <0.0001 and 0.001). In the analysis of tracking the pulse input, the highest torque stiffness, 95 mNm/°, is responsible for the smallest position error, 6.102 ± 5.117°, despite the applied torque acting as compulsory stimuli.

The thesis focuses on applying deep learning and reinforcement learning techniques on human keypoint detection and robot arm manipulation. Inspired by Semi-Autonomous Victim Extraction Robot (SAVER), an autonomous first-aid airway-management robotic system designed to perform Cricothyrotomy on patients is proposed. Perception, decision-making, and control are embedded in the system. In this system, first, the location of the cricothyroid membrane (CTM)-the incision site of Cricothyrotomy- is detected; then, the robot arm is controlled to reach the detected position on a medical manikin. A hybrid neural network (HNNet) that can balance both speed and accuracy is proposed. HNNet is an ensemble-based network architecture that consists of two ensembles: the region proposal ensemble and the keypoint detection ensemble. This architecture can maintain the original high resolution of the input image without heavy computation and can meet the high-precision and real-time requirements at the same time. A dataset containing more than 16,000 images from 13 people, with a clear view of the neck area, and with CTM position labeled by a medical expert was built to train and validate the proposed model. It achieved a success rate of $99.6%$ to detect the position of the CTM with an error of less than 5mm. The robot arm manipulator was trained with the reinforcement learning model to reach the detected location. Finally, the detection neural network and the manipulation process are combined as an integrated system. The system was validated in real-life experiments on a human-sized medical manikin using a Kinect V2 camera and a MICO robot arm manipulator. / Master of Science / The thesis focuses on applying deep learning and reinforcement learning techniques on human keypoint detection and robot arm manipulation. Inspired by Semi-Autonomous Victim Extraction Robot (SAVER), an autonomous first-aid airway-management robotic system designed to perform Cricothyrotomy on patients is proposed. Perception, decision-making, and control are embedded in the system. In this system, first, the location of the cricothyroid membrane(CTM)-the incision site of Cricothyrotomy- is detected; then, the robot arm is controlled to reach the detected position on a medical manikin. A hybrid neural network (HNNet) that can balance both speed and accuracy is proposed. HNNet is an ensemble-based network architecture that consists of two ensembles: the region proposal ensemble and the keypoint detection ensemble. This architecture can maintain the original high resolution of the input image without heavy computation and can meet the high-precision and real-time requirements at the same time. Finally, the detection neural network and the manipulation process are combined as an integrated system. The robot arm manipulator was trained with the reinforcement learning model to reach the detected location. The system was validated in real-life experiments on a human-sized medical manikin using an RGB-D camera and a robot arm manipulator.

This thesis presents a framework for a mobile robot to escort a human to their destination successfully and efficiently. The proposed technique uses accurate path prediction incorporating human intention to locate the robot in front of the human while walking. Human intention is inferred by the head pose, an effective past-proven implicit indicator of intention, and fused with conventional physics-based motion prediction. The human trajectory is estimated and predicted using a particle filter because of the human's nonlinear and non-Gaussian behavior, and the robot control action is determined from the predicted human pose allowing for anticipative autonomous escorting. Experimental analysis shows that the incorporation of the proposed human intention model reduces human position prediction error by approximately 35% when turning. Furthermore, experimental validation with an omnidirectional mobile robotic platform shows escorting up to 50% more accurate compared to the conventional techniques, while achieving 97% success rate. / Master of Science / This thesis presents a method for a mobile robot to escort a human to their destination successfully and efficiently. The proposed technique uses human intention to predict the walk path allowing the robot to be in front of the human while walking. Human intention is inferred by the head direction, an effective past-proven indicator of intention, and is combined with conventional motion prediction. The robot motion is then determined from the predicted human position allowing for anticipative autonomous escorting. Experimental analysis shows that the incorporation of the proposed human intention reduces human position prediction error by approximately 35% when turning. Furthermore, experimental validation with an mobile robotic platform shows escorting up to 50% more accurate compared to the conventional techniques, while achieving 97% success rate. The unique escorting interaction method proposed has applications such as touch-less shopping cart robots, exercise companions, collaborative rescue robots, and sanitary transportation for hospitals.

A collaborative robot is something of growing interest for companies in the manufacturing industries to implement. However, a collaborative robot is quite new in today’s market. An issue that arises is that no implementation process for collaborative robots exists today, as well as no requirement guide for skills, as well as actors, has been defined. The aim of this project was to examine how an implementation process of collaborative robots in manufacturing companies could look like. Focusing on charting the integration process steps of a collaborative robot, and identifying the actors as well as skills needed for successful cobot integration, with the aim achieve the goal of this thesis by answering the research questions. The thesis had the following research questions: Research Question 1 – How is an integration process for implementing a cobot represented in the manufacturing companies? Research Question 2 – What particular skills as well as actors are required when implementing in a cobot in the manufacturing companies? To answer the research questions, the author conducted several interviews with different companies. The interview questions were mainly constructed in order to answer the RQs but also to get an understanding for the different aspects of what a cobot is, what is required as well as how it compares to a traditional industrial robot. The thesis resulted in an implementation process with several steps constructed in order to implement a cobot as well as different aspects of what skills and actors are needed. In order to separate the aspects, the respondents were categorized into different roles which are the developer, integrator and the user. The different roles were all vital, providing an understanding from different perspectives. Keywords: Collaborative robot, cobot, Human-robot interaction, Human-Robot Collaboration, Development strategies, Automation, Industry 4.0.

In der vorliegenden Dissertation wurde ein datengetriebenes Verfahren zum maschinellen Lernen von Mensch-Roboter Interaktionen auf Basis von Mensch-Mensch Demonstrationen entwickelt. Während einer Trainingsphase werden Bewegungen zweier Interakteure mittels Motion Capture erfasst und in einem Zwei-Personen Interaktionsmodell gelernt. Zur Laufzeit wird das Modell sowohl zur Erkennung von Bewegungen des menschlichen Interaktionspartners als auch zur Generierung angepasster Roboterbewegungen eingesetzt. Die Leistungsfähigkeit des Ansatzes wird in drei komplexen Anwendungen evaluiert, die jeweils kontinuierliche Bewegungskoordination zwischen Mensch und Roboter erfordern. Das Ergebnis der Dissertation ist ein Lernverfahren, das intuitive, zielgerichtete und sichere Kollaboration mit Robotern ermöglicht.

In zahlreichen Forschungsprojekten wird unter Einsatz großer finanzieller und personeller Ressourcen daran gearbeitet, dass Roboter die Fabrikhallen verlassen und Teil von Alltagswelten wie Krankenhäusern, Kindergärten und Privatwohnungen werden. Die Konstrukteurinnen und Konstrukteure stehen dabei vor einer nicht-trivialen Herausforderung: Sie müssen die Ambivalenzen und Kontingenzen alltäglicher Interaktion in die diskrete Sprache der Maschinen übersetzen. Wie sie dieser Herausforderung begegnen, welche Muster und Lösungen sie heranziehen und welche Implikationen für die Verwendung von Sozialrobotern dabei gelegt werden, ist der Gegenstand des Buches. Auf der Suche nach der Antwort, was Roboter sozial macht, hat Andreas Bischof Forschungslabore und Konferenzen in Europa und Nordamerika besucht und ethnografisch erforscht. Zu den wesentlichen Ergebnissen dieser Studie gehört die Typologisierung von Forschungszielen in der Sozialrobotik, eine epistemische Genealogie der Idee des Roboters in Alltagswelten, die Rekonstruktion der Bezüge zu 'echten' Alltagswelten in der Sozialrobotik-Entwicklung und die Analyse dreier Gattungen epistemischer Praktiken, derer sich die Ingenieurinnen und Ingenieure bedienen, um Roboter sozial zu machen.:EINLEITUNG 1. WAS IST SOZIALROBOTIK? 1.1 Roboter & Robotik zum Funktionieren bringen 1.2 Drei Problemdimensionen der Sozialrobotik 1.3 Forschungsstand Sozialrobotik 1.4 Problemstellung – Sozialrobotik als „wicked problem“ 2. FORSCHEN, TECHNISIEREN UND ENTWERFEN 2.1 Wissenschaft als (soziale) Praxis 2.2 Technisierung und Komplexitätsreduktion in Technik 2.3 Entwurf, Technik, Nutzung – Technik zwischen Herstellungs- und Wirkungszusammenhang 2.4 Sozialrobotik als Problemlösungshandeln 3. METHODOLOGIE UND METHODEN DER STUDIE 3.1 Forschungsstil Grounded Theory 3.2 Ethnografie und narrative Experteninterviews 3.3 Auswertungsmethoden und Generalisierung 3.4 Zusammenfassung 4. DER ROBOTER ALS UNIVERSALWERKZEUG 4.1 Roboter als fiktionale Apparate 4.2 Robotik als Lösungsversprechen 4.3 Computer Science zwischen Wissenschaft und Design 4.4 Fazit – Das Erbe des Universalwerkzeugs 5. FORSCHUNGS- UND ENTWICKLUNGSZIELE DER SOZIALROBOTIK 5.1 Bedingungen projektförmiger Forschung 5.2 Dimensionen und Typen der Ziele von Sozialrobotik 5.3 Beschreibung der Typen anhand der Verteilung der Fälle 5.4 Ko-Konstruktion der Anwendung an Fallbeispielen 5.5 Fazit – Typen von Sozialität in Entwicklungszielen 6. EPISTEMISCHE PRAKTIKEN UND INSTRUMENTE DER SOZIALROBOTIK 6.1 Praktiken der Laboratisierung des Sozialen 6.2 Alltägliche und implizite Heuristiken 6.3 Inszenierende Praktiken 6.4 Fazit – Wechselspiele des Erzeugens und Beobachtens 7. FAZIT 7.1 Phänomenstruktur der Sozialrobotik 7.2 Entwicklung als Komplexitätspendel 7.3 Methodologischer Vorschlag für den Entwicklungsprozess

L'interaction homme-robot arrive dans une phase intéressante ou la relation entre un homme et un robot est envisage comme 'un partenariat plutôt que comme une simple relation maitre-esclave. Pour que cela devienne une réalité, le robot a besoin de comprendre le comportement humain. Il ne lui suffit pas de réagir de manière appropriée, il lui faut également être socialement proactif. Pour que ce comportement puis être mise en pratique le roboticien doit s'inspirer de la littérature déjà riche en sciences sociocognitives chez l'homme. Dans ce travail, nous allons identifier les éléments clés d'une telle interaction dans le contexte d'une tâche commune, avec un accent particulier sur la façon dont l'homme doit collaborer pour réaliser avec succès une action commune. Nous allons montrer l'application de ces éléments au cas un système robotique afin d'enrichir les interactions sociales homme-robot pour la prise de décision. A cet égard, une contribution a la gestion du but de haut niveau de robot et le comportement proactif est montre. La description d'un modèle décisionnel d'collaboration pour une tâche collaboratif avec l'humain est donnée. Ainsi, l'étude de l'interaction homme robot montre l'intéret de bien choisir le moment d'une action de communication lors des activités conjointes avec l'humain / Human Robot Interaction is entering into the interesting phase where the relationship with a robot is envisioned more as one of companionship with the human partner than a mere master-slave relationship. For this to become a reality, the robot needs to understand human behavior and not only react appropriately but also be socially proactive. A Companion Robot will also need to collaborate with the human in his daily life and will require a reasoning mechanism to manage thecollaboration and also handle the uncertainty in the human intention to engage and collaborate. In this work, we will identify key elements of such interaction in the context of a collaborative activity, with special focus on how humans successfully collaborate to achieve a joint action. We will show application of these elements in a robotic system to enrich its social human robot interaction aspect of decision making. In this respect, we provide a contribution to managing robot high-level goals and proactive behavior and a description of a coactivity decision model for collaborative human robot task. Also, a HRI user study demonstrates the importance of timing a verbal communication in a proactive human robot joint action

An important capacity that is still lacking in intelligent systems such as robots, is the ability to use concepts in a human-like manner. Indeed, the use of concepts has been recognised as being fundamental to a wide range of cognitive skills, including classification, reasoning and memory. Intricately intertwined with language, concepts are at the core of human cognition; but despite a large body or research, their functioning is as of yet not well understood. Nevertheless it remains clear that if intelligent systems are to achieve a level of cognition comparable to humans, they will have to posses the ability to deal with the fundamental role that concepts play in cognition. A promising manner in which conceptual knowledge can be acquired by an intelligent system is through ongoing, incremental development. In this view, a system is situated in the world and gradually acquires skills and knowledge through interaction with its social and physical environment. Important in this regard is the notion that cognition is embodied. As such, both the physical body and the environment shape the manner in which cognition, including the learning and use of concepts, operates. Through active partaking in the interaction, an intelligent system might influence its learning experience as to be more effective. This work presents experiments which illustrate how these notions of interaction and embodiment can influence the learning process of artificial systems. It shows how an artificial agent can benefit from interactive learning. Rather than passively absorbing knowledge, the system actively partakes in its learning experience, yielding improved learning. Next, the influence of embodiment on perception is further explored in a case study concerning colour perception, which results in an alternative explanation for the question of why human colour experience is very similar amongst individuals despite physiological differences. Finally experiments, in which an artificial agent is embodied in a novel robot that is tailored for human-robot interaction, illustrate how active strategies are also beneficial in an HRI setting in which the robot learns from a human teacher.

Au cours des dernières années, nous avons assisté à un changement de paradigme, passant de la fabrication de robots rigides à des robots compliants. Ceci est dû à plusieurs raisons telles que l'amélioration de l'efficacité des robots dans la réalisation des mouvements explosifs ou cycliques. En fait, l'une des premières motivations à l'origine de ce changement est la sécurité. Parlant de la sécurité à la fois du sujet humain et du robot, tout en s'engageant dans des tâches collaboratives. Ainsi la désignation des cobots. Les cobots peuvent aider un opérateur humain expérimenté dans plusieurs domaines où la précision est essentielle, comme les applications industrielles ou les tâches médicales. Jusqu'à présent, les cobots présentent toujours des problèmes de sécurité, même avec des recommandations réglementaires telles que ISO / TS 15066 et ISO 10218-1 et 2 qui limitent leurs avantages économiques. Dans cette vue, plusieurs projets de recherche ont été lancés dans le monde entier pour améliorer la dynamique des cobots par rapport à la sécurité, ANR-SISCob (Safety Intelligent Sensor for cobots) étant l'un de ces projets. Les travaux menés au cours de cette thèse ont pour but de concevoir des dispositifs de sécurité qui sécuriseront les robots en y introduisant l’aspect de compliance. En effet, nous avons développé deux dispositifs dans lesquels l'aspect sécurité est atteint avec deux approches différentes :- Prismatic Compliant Joint (PCJ) : qui vise à la mise en œuvre dans les articulations linéaires, car peu de travaux ont traité de tels systèmes d'actionnement. Ici, la sécurité est atteinte biomimétiquement tout en faisant face à d'autres critères de sécurité liés aux propriétés mécaniques du corps humain.- Variable Stiffness Safety Oriented Mechanism (V2SOM) : Contrairement au premier dispositif d'inspiration biomimétique qui sert aux systèmes d'actionnement linéaires, le profil de sécurité du V2SOM est axé sur la sécurité selon deux critères de sécurité: force d’impact et HIC. L'aspect ‘orienté sécurité’ est dû à ce que nous appelons la capacité de découplage d'inertie de son profil de rigidité. V2SOM est actuellement dans ses dernières étapes de brevetage.Ces deux appareils seront intégrés dans un robot sériel réalisé dans notre laboratoire. / In the recent years, we witnessed a paradigm shift from making stiff robots toward compliant ones. This is due to several reasons such as enhancing the efficiency of robots in making explosive or cyclic motion. In fact, one of the earliest motivations from which this change stems are safety. Speaking of safety of both the human subject and the robot alike, while engaging in a collaborative task. Thus, the designation of cobots. Cobots may assist well-experienced human operator in several domains where precision is a must, such as industrial applications or medical tasks. Until now cobots still display safety concerns, even with regulatory recommendations such as ISO/TS 15066 and ISO 10218-1 et 2 that limits their economic benefits. In this view, several research projects were launched worldwide to enhance the cobot’s dynamics vs safety, ANR-SISCob (Safety Intelligent Sensor for cobots) is one of these projects. The works conducted during this thesis aims at making safety devices that will make robots safe by introducing compliance aspect in them. Indeed, we developed two devices in which safety aspect is achieved with two different approaches: - Prismatic Compliant Joint (PCJ): is aimed at prismatic joint’s implementation, as few works have dealt with such actuation systems. Herein, safety is biomimetically attained while coping with other safety criteria related to the mechanical properties of human body. - Variable Stiffness Safety Oriented Mechanism (V2SOM): Unlike the first device that’s biomimetically inspired and serves at linear actuation systems, V2SOM’s safety profile is safety oriented according to two safety criteria Impact force and HIC, and is designed for rotary actuation. The safety oriented aspect is due to what we call inertia decoupling capacity of its stiffness profile. V2SOM is currently in its final patenting process.Both devices will be integrated in serial robot built in our lab.