In this thesis, we propose and present a multimodal interaction system that can provide a natural way for human-computer interaction. The core idea of this system is to help users to interact with the machine naturally by recognizing various gestures from the user from a wearable device. To achieve this goal, we have implemented a system including both hardware solution and gesture recognizing approaches. For the hardware solution, we designed and implemented a data glove based interaction device with multiple kinds of sensors to detect finger formations, touch commands and hand postures. We also modified and implemented two gesture recognizing approach based on support vector machine (SVM) as well as the lookup table. The detailed design and information is presented in this thesis. In the end, the system achieves supporting over 30 kinds of touch commands, 18 kinds of finger formation, and 10 kinds of hand postures as well as the combination of finger formation and hand posture with the recognition rate of 86.67% as well as the accurate touch command detection. We also evaluated the system from the subjective user experience.

none

Alors que les systèmes de reconnaissance gestuelle actuels privilégient souvent un usage intérieur, nous nous intéressons à la conception d'un système dont l'utilisation est possible en environnement extérieur et en mobilité. Notre objectif est le contrôle temps-réel d'un robot mobile dont l'usage est destiné aux fantassins débarqués. La contribution principale de cette thèse est le développement d'une chaîne de reconnaissance gestuelle temps réel, qui peut être entraînée en quelques minutes avec: un faible nombre d'exemples ("small data"); des gestes choisis par l'utilisateur; une résilience aux gestes mal réalisés; ainsi qu'une faible empreinte CPU. Ceci est possible grâce à deux innovations clés: d'une part, une technique pour calculer des distances entre séries temporelles en flux, basée sur DTW; d'autre part, une rétro-analyse efficace du flux d'apprentissage afin de déterminer les hyperparamètres du modèle sans intervention de l'utilisateur. D'autre part, nous avons construit notre propre gant de données et nous l'utilisons pour confirmer expérimentalement que la solution de reconnaissance gestuelle permet le contrôle temps réel d'un robot en mobilité. Enfin, nous montrons la flexibilité de notre technique en ce sens qu'elle permet de contrôler non seulement des robots, mais aussi des systèmes de natures différentes. / Although gesture recognition has been studied for several decades, much research stays in the realm of indoors laboratory experiments. In this thesis, we address the problem of designing a truly usable, real- world gesture recognition system, focusing mainly on the real-time control of an outdoors robot for use by military soldiers. The main contribution of this thesis is the development of a real-time gesture recognition pipeline, which can be taught in a few minutes with: very sparse input ("small data"); freely user-invented gestures; resilience to user mistakes during training; and low computation requirements. This is achieved thanks to two key innovations: first, a stream-enabled, DTW-inspired technique to compute distances between time series; and second, an efficient stream history analysis procedure to automatically determine model hyperparameters without user intervention. Additionally, a custom, hardened data glove was built and used to demonstrate successful gesture recognition and real-time robot control. We finally show this work's flexibility by furthermore using it beyond robot control to drive other kinds of controllable systems.

Vast improvements in robotics and wireless communication have made teleoperated robots significantly more prevalent in industry, defense, and research. To help bridge the gap for these robots in the workplace, there has been a tremendous increase in research toward the development of biomimetic robotic hands that can simulate human operators. However, current methods of control are limited in scope and do not adequately represent human muscle memory and skills. The vision of this thesis is to provide a pathway for overcoming these limitations and open an opportunity for development and implementation of a cost effective methodology towards controlling a robotic hand. The first chapter describes the experiments conducted using Flexpoint bend sensors in conjunction with a simple voltage divider to generate a cost-effective data glove that is significantly less expensive than the commercially available alternatives. The data glove was able to provide sensitivity of less than 0.1 degrees. The second chapter describes the molding process for embedding pressure sensors in silicone skin and data acquisition from them to control the robotic hand. The third chapter describes a method for parsing and observing the information from the data glove and translating the relevant control variables to the robotic hand. The fourth chapter focuses on the feasibility of the brain computer interfaces (BCI) and successfully demonstrates the implementation of a simple brain computer interface in controlling a robotic hand. / Master of Science

For over 30 years, the development and control of anthropomorphic robotic hands has been a highly popular sub-discipline in robotics research. Because the human hand is an extremely sophisticated system, both in its mechanical and sensory abilities, engineers have been fascinated with replicating these abilities in artificial systems. The applications of robotic hands typically fall under the categories of standalone testbed platforms, mostly to conduct research on manipulation, prosthetics, and robotic end effectors for larger systems. The teleoperation of robotic hands is another application with significant potential, where users can control a manipulator in real time to accomplish diverse tasks. In controlling a device that seeks to emulate the function of the human hand, it is intuitive to choose a human-machine interface (HMI) that will allow for the most intuitive control. Data gloves are the ideal HMI for this need, allowing a robotic hand to accurately mimic the human operator's natural movements. In this paper we present a combined review on the critical design aspects of data gloves and robotic hands. In literature, many of the proposed designs covering both these topical areas, robotic hand and data gloves, are cost prohibitive which limits their implementation for intended tasks. After reviewing the literature, new designs of robotic hand and data glove technology are also presented, introducing low cost solutions that can serve as accessible platforms for researchers, students, and engineers to further the development of teleoperative applications. / Master of Science / For over 30 years, the development and control of anthropomorphic robotic hands has been a highly popular sub-discipline in robotics research. Because the human hand is an extremely sophisticated system, both in its mechanical and sensory abilities, engineers have been fascinated with replicating these abilities in artificial systems. The applications of robotic hands typically fall under the categories of standalone testbed platforms, mostly to conduct research on manipulation, prosthetics, and robotic end effectors for larger systems. The teleoperation of robotic hands is another application with significant potential, where users can control a manipulator in real time to accomplish diverse tasks. In controlling a device that seeks to emulate the function of the human hand, it is intuitive to choose a human-machine interface (HMI) that will allow for the most intuitive control. Data gloves are the ideal HMI for this need, allowing a robotic hand to accurately mimic the human operator’s natural movements. In this paper we present a combined review on the critical design aspects of data gloves and robotic hands. In literature, many of the proposed designs covering both these topical areas, robotic hand and data gloves, are cost prohibitive which limits their implementation for intended tasks. After reviewing the literature, new designs of robotic hand and data glove technology are also presented, introducing low cost solutions that can serve as accessible platforms for researchers, students, and engineers to further the development of teleoperative applications.

The purpose of the thesis was to create a data glove that can translate ASL by reading the finger- and hand movements. Furthermore, the applicability of conductive fabric as stretch sensors was explored. To read the hand gestures stretch sensors constructed from conductive fabric were attached to each finger of the glove to distinguish how much they were bent. The hand movements were registered using a 3-axis accelerometer which was mounted on the glove. The sensor values were read by an Arduino Nano 33 IoT mounted to the wrist of the glove which processed the readings and translated them into the corresponding sign. The microcontroller would then wirelessly transmit the result to another device through Bluetooth Low Energy. The glove was able to correctly translate all the signs of the ASL alphabet with an average accuracy of 93%. It was found that signs with small differences in hand gestures such as S and T were harder to distinguish between which would result in an accuracy of 70% for these specific signs. / Syftet med uppsatsen var att skapa en datahandske som kan översätta ASL genom att läsa av finger- och handrörelser. Vidare undersöktes om ledande tyg kan användas som sträcksensorer. För att läsa av handgesterna fästes ledande tyg på varje finger på handsken för att urskilja hur mycket de böjdes. Handrörelserna registrerades med en 3-axlig accelerometer som var monterad på handsken. Sensorvärdena lästes av en Arduino Nano 33 IoT monterad på handleden som översatte till de motsvarande tecknen. Mikrokontrollern överförde sedan resultatet trådlöst till en annan enhet via Bluetooth Low Energy. Handsken kunde korrekt översätta alla tecken på ASL-alfabetet med en genomsnittlig exakthet på 93%. Det visade sig att tecken med små skillnader i handgester som S och T var svårare att skilja mellan vilket resulterade i en noggrannhet på 70% för dessa specifika tecken.