The thesis focuses on applying deep learning and reinforcement learning techniques on human keypoint detection and robot arm manipulation. Inspired by Semi-Autonomous Victim Extraction Robot (SAVER), an autonomous first-aid airway-management robotic system designed to perform Cricothyrotomy on patients is proposed. Perception, decision-making, and control are embedded in the system. In this system, first, the location of the cricothyroid membrane (CTM)-the incision site of Cricothyrotomy- is detected; then, the robot arm is controlled to reach the detected position on a medical manikin. A hybrid neural network (HNNet) that can balance both speed and accuracy is proposed. HNNet is an ensemble-based network architecture that consists of two ensembles: the region proposal ensemble and the keypoint detection ensemble. This architecture can maintain the original high resolution of the input image without heavy computation and can meet the high-precision and real-time requirements at the same time. A dataset containing more than 16,000 images from 13 people, with a clear view of the neck area, and with CTM position labeled by a medical expert was built to train and validate the proposed model. It achieved a success rate of $99.6%$ to detect the position of the CTM with an error of less than 5mm. The robot arm manipulator was trained with the reinforcement learning model to reach the detected location. Finally, the detection neural network and the manipulation process are combined as an integrated system. The system was validated in real-life experiments on a human-sized medical manikin using a Kinect V2 camera and a MICO robot arm manipulator. / Master of Science / The thesis focuses on applying deep learning and reinforcement learning techniques on human keypoint detection and robot arm manipulation. Inspired by Semi-Autonomous Victim Extraction Robot (SAVER), an autonomous first-aid airway-management robotic system designed to perform Cricothyrotomy on patients is proposed. Perception, decision-making, and control are embedded in the system. In this system, first, the location of the cricothyroid membrane(CTM)-the incision site of Cricothyrotomy- is detected; then, the robot arm is controlled to reach the detected position on a medical manikin. A hybrid neural network (HNNet) that can balance both speed and accuracy is proposed. HNNet is an ensemble-based network architecture that consists of two ensembles: the region proposal ensemble and the keypoint detection ensemble. This architecture can maintain the original high resolution of the input image without heavy computation and can meet the high-precision and real-time requirements at the same time. Finally, the detection neural network and the manipulation process are combined as an integrated system. The robot arm manipulator was trained with the reinforcement learning model to reach the detected location. The system was validated in real-life experiments on a human-sized medical manikin using an RGB-D camera and a robot arm manipulator.

No description available.

L'analyse de la marche et la mesure du déplacements des articulations humaines ont été largement étudiées. Les artefacts de tissus «mous» sont une source fréquente d'erreur pour la plupart des méthodes de mesure utilisées. La procédure standard en analyse de la marche consiste à utiliser une combinaison de mesures pour l'estimation efficace des angles articulaires et de la position des segments du corps humain. Ce travail propose le développement d'un système d'analyse de la marche reposant sur un robot parallèle à câbles équipé de plusieurs capteurs mesurant spécifiquement les déplacements du genou. Nous considérons le cas général pour lequel les articulations humaines se comportent comme des joints à 6 degrés de liberté reliant deux segments du corps. Afin de déterminer la position et l'orientation de ces segments, 14 câbles y sont attachés, ce qui permet de considérer ces segments comme les organes effecteurs de robots parallèles. Leur position peut alors être calculée à partir de la mesure de la longueur des câbles. Cependant, ces mesures sont entachées de bruit à cause des artefacts de tissus «mous». Afin d'améliorer la précision des résultats, le système propose aussi l'utilisation d'autres capteurs de nature différente : plusieurs capteurs inertiels (avec accéléromètres et gyroscopes), un système de motion capture, des capteurs de pression plantaire, des capteurs de distance (IR et résistance variable) et des capteurs de force pour mesurer la contraction musculaire. Plusieurs approches globales sont disponibles pour l'analyse du genou lors de la marche. Les choix technologiques effectués impactent directement sur la conception de notre système et imposent le développement de matériel spécifique pour mener à bien les mesures, tel que le collier flexible utilisé d'une part pour permettre l'attache des câbles sur les segments du patient et d'autres part pour supporter les capteurs supplémentaires. Nous traitons le collier comme une chaîne cinématique sérielle et nous proposons une méthode d'étalonnage qui ne nécessite pas d'utiliser les mesures angulaires des articulations contrairement aux méthodes existantes. Nous décrivons le protocole expérimental ainsi que les méthodes utilisées pour synchroniser les données issues de plusieurs ordinateurs. Les données sont ensuite fusionnées pour obtenir la pose du collier et donc celle des segments du patient. Enfin, ce travail permet d'identifier les modifications à apporter au système pour une meilleure analyse de la marche, ce qui pourra servir de base à un système de rééducation complet. / Gait analysis and human joint motion measurement has been studied extensively in the recent past. In order to address the effects of soft tissue artifacts (STA), a common source of error in most type of measurements, the standard procedure in gait analysis has been to use a combination of measurement methods for efficient estimation of joint angles and the body segment poses. This work proposes a gait analysis system based on a multi-sensor cable-driven parallel manipulator, focusing specifically on tracking the human knee. Our system assumes a human joint to be a general 6 DOF joint between 2 body segments. In order to measure pose of these body segments, up to 14 wires are attached to these human body segments and this permits the system to treat each of these body segments as the end-effector of a parallel mechanism. The pose of the body segments can thus be determined by measuring the wire lengths and solving the forward kinematics of this parallel architecture. The system is also equipped to use additional sensors including inertial sensors (accelerometers and gyroscopes), a 12 camera optical tracking system, in-shoe pressure sensors, variable length resistive wires, IR distance sensors, force sensors to measure muscle contraction. A number of choices are available in the approach for analyzing the knee during gait activity and the design of the setup depends on these choices. This work discuses the options available and details how they have impacted the choices we make in developing the experimental setup. We discuss the hardware developed and used, and specifically discuss the flexible collar used to attach wires to the patient body and to hold the additional sensors. We treat the collar as a serial kinematic chain and propose a calibration method for it that, unlike commonly used calibration techniques, avoids using joint angle measurements. We then outline the experiment and the methods used to synchronize and fuse the data from all sensors to obtain a pose estimate for the collar and thus, the body segments. Finally, this work helps identify steps necessary to improve the current setup and lays the groundwork for a complete rehabilitation system.

No description available.