Uncalibrated visual servoing (VS) can improve robot performance without needing camera and robot parameters. Multiple cameras improve uncalibrated VS precision, but no works exist simultaneously using more than two cameras. The first data for uncalibrated VS simultaneously using more than two cameras are presented. VS performance is also compared for two different camera models: a high-cost camera and a low-cost camera, the difference being image noise magnitude and focal length. A Kalman filter based control law for uncalibrated VS is introduced and shown to be stable under the assumptions that robot joint level servo control can reach commanded joint offsets and that the servoing path goes through at least one full column rank robot configuration. Adaptive filtering by a covariance matching technique is applied to achieve automatic camera weighting, prioritizing the best available data. A decentralized sensor fusion architecture is utilized to assure continuous servoing with camera occlusion. The decentralized adaptive Kalman filter (DAKF) control law is compared to a classical method, Gauss-Newton, via simulation and experimentation. Numerical results show that DAKF can improve average tracking error for moving targets and convergence time to static targets. DAKF reduces system sensitivity to noise and poor camera placement, yielding smaller outliers than Gauss-Newton. The DAKF system improves visual servoing performance, simplicity, and reliability.

Orientador: Marconi Kolm Madrid / Tese (doutorado) - Universidade Estadual de Campinas, Faculdade de Engenharia Eletrica e de Computação / Made available in DSpace on 2018-08-15T23:54:05Z (GMT). No. of bitstreams: 1 Nicolato_Fabricio_D.pdf: 1516280 bytes, checksum: 96229803f3bca54f669d4dcc22108c02 (MD5) Previous issue date: 2007 / Resumo: Esta tese trata o problema da resolução do modelo cinemático inverso para manipuladores industriais redundantes ou não. O problema foi abordado por um método de busca heurística no qual a solução da cinemática inversa é construída passo a passo calculando-se a contribuição do movimento de apenas uma junta a cada iteração. Dessa forma, o problema n-dimensional é transformado em problemas unidimensionais mais simples, cuja solução analítica tanto para juntas rotacionais quanto para juntas prismáticas é apresentada em termos da representação de Denavit-Hartenberg. O método proposto não possui singularidades internas. Além disso, o método foi expandido para incorporar informações de sensores externos visando fazer com que o processo seja mais robusto a incertezas nas modelagens envolvidas. Foram realizadas diversas simulações e comparações com técnicas tradicionais que evidenciaram as vantagens da abordagem proposta. O trabalho também englobou o projeto e a construção de um ambiente experimental e a implementação das técnicas desenvolvidas na parte teórica. Desenvolveu-se um sistema com um robô planar redundante de 3 DOF, assim como seus sistemas de controle, acionamento e interfaceamento usando técnicas de sistemas hardware-inthe-loop e lógica programável. As técnicas desenvolvidas foram aplicadas no ambiente experimental demonstrando características como: facilidade de lidar com redundâncias, capacidade de resolução em tempo real, robustez a incertezas de parâmetros etc / Abstract: This thesis deals with the problem of solving the inverse kinematics model of redundant and nonredundant industrial manipulators. The work was developed in a theoretical and a practical part. The problem was approached by an heuristic search method in which the solution of the inverse kinematics is built step by step calculating the movement contribution of just a single joint for each iteration. In that way, the n-dimensional problem is transformed in simpler one-dimensional problems, whose analytic solution for both rotational joints and prismatic joints is presented in terms of the Denavit and Hartenberg representation. The proposed method does not possess internal singularities. Furthermore, the method was expanded to incorporate information of external sensor in order to make the process more robust to uncertainties in the involved modelings. Several results of simulations and comparisons with traditional techniques, which evidence the advantages of the proposed approach, are presented. The work also included the construction of an experimental environment and the implementation of the techniques developed in the theoretical part. The details of a system with a 3-DOF redundant robot as well as its control system, drivers and interfaces using hardware-in-theloop techniques and programmable logic are presented. The developed techniques were applied in the experimental environment are demonstrating their efficiency and evidencing characteristics like: easiness of dealing with redundancies, real time capacity, robustness for parameters uncertainties etc / Doutorado / Automação

L’autonomie d’un agent mobile se définit par sa capacité à naviguer dans un environnement sans intervention humaine. Cette tâche s’avère très demandée pour les robots d’assistance à la personne. C’est pour cela que notre contribution s’est portée en particulier sur l’instrumentation et l’augmentation de l’autonomie d’un fauteuil roulant pour les personnes à mobilité réduite. L’objectif de ce travail est de concevoir des lois de commande qui permettent à un robot de naviguer en temps réel et en toute autonomie dans un environnement inconnu. Un cadre de perception virtuelle unifié est introduit et permet de projeter l’espace navigable obtenu par des observations éventuellement multiples. Une approche de navigation autonome et sûre a été conçue pour se déplacer dans un environnement peu encombré dont la structure peut être assimilée à un couloir (lignes au sol, murs, délimitation herbes, routes...). La problématique a été résolue en utilisant le formalisme de l’asservissement visuel. Les caractéristiques visuelles utilisées dans la loi de commande ont été construites à partir de la représentation virtuelle (à savoir la position du point de fuite et l’orientation de la ligne médiane du couloir). Pour assurer une navigation sûre et lisse, même lorsque ces paramètres ne peuvent pas être extraits, nous avons conçu un observateur d’état pour estimer les caractéristiques visuelles dans le but de maintenir la commande fonctionnelle du robot. Cette approche permet de faire naviguer un robot mobile dans un couloir même en cas de défaillance sensorielle (données non fiables) et/ou de perte de mesure. La première contribution de cette thèse a été étendue en traitant tout type d’environnement encombré statique ou dynamique. Cela a été réalisé en utilisant le diagramme de Voronoï. Le diagramme de Voronoï généralisé, également appelé squelette, est une représentation puissante de l’environnement. Il définit un ensemble de chemins à la distance maximale des obstacles. Dans ce travail, une approche d’asservissement visuel basée sur le squelette extrait en temps réel était proposée pour une navigation autonome et sûre des robots mobiles. La commande est basée sur une approximation du DVG local en utilisant le Delta Medial axis, un algorithme de squelettisation rapide et robuste. Ce dernier produit un squelette filtré de l’espace libre entourant le robot en utilisant un paramètre qui prend en compte la taille du robot. Cette approche peut faire face aux bruits de mesure au niveau de la perception et au niveau de la commande à cause des glissement des roues. C’est pour cela que nous avons conçu une approche d’asservissement visuel sur une prédiction d’une linéarisation du DVG. Une analyse complète a été réalisée pour montrer la stabilité des lois de commandes proposées. Des simulations et des tests expérimentaux valident l’approche proposée. / The autonomy of a mobile agent is defined by its ability to navigate in an environment without human intervention. This task is very required for personal assistance robots. That’s why our contribution has been particularly focused on instrumentation and increasing the autonomy of a wheelchair for reduced mobility peaple. The objective of this work is to design control laws that allow a robot to navigate in real time and independently in an unknown environment. A unified virtual perception framework is introduced and allows to project the navigable space obtained by possibly multiple observations. First we designed an autonomous and safe navigation approach in environment whose structure can be assimilated to a corridor (lines on the ground, walls, delimitation of grasses, roads ...). We have solved this problem by using the formalism of visual servoing. The visual characteristics used in the control law were constructed from the virtual representation (ie the position of the vanishing point and the orientation of the center line of the corridor). To ensure safe and smooth navigation, even when these parameters can not be extracted, we have designed a finite-time state observer to estimate the visual characteristics in order to maintain the robot’s control efficient. This approach let a mobile robot navigate in a corridor even in in the case of sensory failure (unreliable data) and/or loss of measurement. We have extended the first contribution of this work with dealing with any type of static or dynamic environment. This was done using the Voronoi diagram. The Generalized Voronoi Diagram (GVD), also named skeleton, is a powerful environment representation, since, among other reasons, it defines a set of paths at maximal distance from the obstacles. In this work, a real time skeleton based visual servoing approach is proposed for a safe autonomous navigation of mobile robots. The control is based on an approximation of the local GVD using the Delta Medial Axis, a fast and robust skeletonization algorithm. The latter produces a filtered skeleton of the free space surrounding the robot using a pruning parameter that takes into account the robot size. This approach can cope with measurement noises at the perception and control with the wheel slip. This is why we have designed a visual servoing approach on a prediction of a GVD linearization. A complete analysis was performed to show the stability of the proposed control laws. Simulations and experimental tests validate the proposed approach.

Les avancées technologiques ont ouvert la voie à des approches de biopsie optique innovantes. Elles permettent à l'inverse des méthodes physiques de profiter des avantages d'une procédure mini-invasive, temps réels et répétitive. Le système de tomographie par cohérence optique (OCT) (la technique de biopsie optique utilisée dans cette thèse) propose des approches qui naviguent dans le corps humain grâce à des sondes endoscopiques robotisées. Toutefois, leur contrôle une fois à l’intérieur du corps devient difficile, surtout si l’objectif est de suivre l’évolution d'une zone cible, en faisant un travail de repositionnement dans le temps. L'asservissement visuel est un outil de choix pour le contrôle et le positionnement directement par l'image. Néanmoins, la richesse des informations présentes dans les images autorisent l'utilisation de plusieurs types d'information visuelle. Dans ce contexte, nous proposons l'utilisation de primitives visuelles innovantes fondées sur les ondelettes. Ainsi, deux approches d'asservissements visuels fondées sur les ondelettes ont été développées. La première approche est un asservissement visuel 2D pose fondé sur les ondelettes spectrales continues qui assure une convergence sur un espace plus important avec une bonne robustesse au bruit et une commande découplée. La deuxième est un asservissement visuel 2D direct fondé sur les ondelettes multirésolution, principalement pour faire du positionnement aux petits déplacements. Par ailleurs, la deuxième méthode couvre les 6 DDL quand la première se limite aux 3 DDL dans les images CCD. De plus, ces deux approches ont prouvé leurs aptitudes à faire du positionnement des coupes OCT. Mais encore, nous avons proposé une méthode de positionnement partitionnée que nous pouvons qualifier d'hybride, car elle exploite deux modalités d'images (OCT - CCD) pour assurer un positionnement sur SE(3) d'un échantillon. De même, nous avons proposé une méthode d'étalonnage des images de coupe et de volume OCT, liée aux distorsions générées par le chemin optique parcouru par le faisceau laser OCT. Finalement, ces travaux ouvrent la voie vers des applications dans le positionnement des volumes OCT, la compensation de mouvement physiologique et le suivi d'outils par des images OCT. / The technological advances have facilitated the optical biopsy approaches, unlike physical methods to take advantage of a minimally invasive, real time and repetitive procedure. The optical coherence tomography system is one of the optical biopsy techniques used in this thesis to prospect in the human body with robotized OCT endoscopic probes. Nevertheless, their control once inside the body becomes difficult, especially if the goal is following changes in the target area. The visual servoing is an ideal tool for the control and positioning of the robot. However, the amount of information present in the images allows the use of several types of visual features. In this thesis, we propose to use an innovative visual servoing feature based on wavelets. This representation developed as the evolution of the Fourier transform for non-stationary signals provides a time-frequency representation of the signal with a better extraction of the relevant information. Indeed, two visual servoing approaches based on wavelets were developed. The first approach is a 2D pose visual servoing based on spectral continuous wavelets, which ensures convergence over a larger area and decoupled control. The second is a direct 2D visual servoing based on multiresolution wavelets, mainly for small displacements positioning. However, the latter covers the 6 DOF when the previous one is limited to 3 DDL with a CCD camera. Both approaches have proven their ability to make the positioning of B-Scan OCT images. After that, we have proposed a method of partitioned positioning, that we can qualify by hybrid because it uses two image modalities to ensure SE(3) positioning of a sample. On the other side, we proposed a calibration method of B-Scan and 3D-Scan OCT images, due to the distortions generated by the optical path of the laser beam in OCT. Finally, these thesis is a beginning work for applications in positioning of 3D-Scan OCT, physiological motion compensation and monitoring tools by OCT images.

On rencontre aujourd'hui la vision par ordinateur employée pour la réalisation de nombreuses applications de la robotique moderne. L'un des axes de recherche actuel qui tend à améliorer ces systèmes est centré sur la commande. L'objectif est de proposer des schémas de commande originaux permettant de lier efficacement les informations mesurées par les capteurs de vision aux actions que l'on souhaite réaliser avec les robots. C'est dans cet aspect que s'inscrit ce document en apportant de nouvelles méthodes à la commande robotique classique faisant intervenir la vision, l'asservissement visuel. Le cas de pertes d'informations visuelles pendant la réalisation d'une tâche d'asservissement visuel est étudié. Dans ce sens, deux méthodes de prédiction qui permettent à la tâche d'être réalisée malgré ces pertes sont présentées. Puis une méthode inédite de correction est proposée. Celle-ci permet d'obtenir de meilleurs résultats de prédiction qu'une méthode classique, comme le démontrent des résultats obtenus en simulation et en condition réelle. Enfin, dans le contexte de la réalisation de plusieurs tâches d'asservissement visuel successives, une nouvelle méthode est présentée. Celle-ci exploite les caractéristiques d'un schéma de commande utilisé depuis quelques dizaines d'années dans l'industrie et la recherche, la commande prédictive basée modèle. Des résultats obtenus en simulation proposent de visualiser les effets de cette méthode sur le comportement d'un drone qui embarque une caméra. / The computer vision is used for the achievement of many applications of modern robotics. One of the current research topics that aims to improve these systems is focused on command. The objective consists to propose original control schemes to effectively link the information measured by the vision sensor to the actions that are to be achieved with the robots. This document is part of this look by bringing new methods to classical robotic control involving vision, the visual servoing.The case of visual information losses during the achievement of a visual servoing task is studied. In this sense, two prediction methods that allow the task to be achieved despite these losses are presented. Then a new method of correction is proposed. This provides better prediction results than a conventional method, as shown by the results obtained in simulation and in real conditions.Finally, in the context of the achievement of several successive visual servoing tasks, a new method is presented. This exploits the characteristics of a control scheme used for several decades in industry and research, model based predictive control. The results obtained in simulation propose to see the effect of this method on the behavior of a drone that embeds a camera.

Ce manuscrit traite de la conception d’un robot quadrirotor bio-inspiré. Ce robot, nommé QuadMorphing, s’inspire de l’oiseau et possède la capacité de se replier en vol afin de diminuer son envergure. Cette particularité est intéressante pour des problématiques d’évitement d’obstacles dans des milieux encombrés.Le travail présenté ici contient une présentation du robot où la plateforme mécatronique y est décrite en détails. Puis, des résultats expérimentaux sont présentés et commentés afin de quantifier les performances du prototype QuadMorphing durant des scénarios de franchissement d’obstacles.La deuxième partie de cette thèse traite de l’estimation de la taille d’obstacles en vol grâce à une perception visuelle monoculaire. Deux algorithmes d’estimation ont été simulés afin d’être validés pour être ensuite mis en place sur une nouvelle version du robot qui a été testée expérimentalement. Ces estimations permettent par la suite de rendre le robot plus autonome pour éviter les collisions avec son environnement et actionner son système de changement de forme si cela est nécessaire. / This paper describes a bio-inspired quadrotor design. This robot, called QuadMorphing, is inspired by birds and has the ability to fold its mechanical structure to reduce its wingspan during the flight. This feature could be useful for obstacle avoidance task in cluttered environments.The work presented here contains a full description of the mechatronic structure. Then, experimental results are presented and discussed in order to quantify the QuadMorphing performances during obstacle avoidance scenarios.The second part of this thesis deals with estimating obstacle size during flight using monocular visual perception. Two estimation algorithms were simulated in order to be validated and then implemented for experimental testing on a new version of the robot. In order to make the robot autonomous, the estimation of the size of the obstacle allows the robot to avoid collisions with its environment and to perform its morphological reduction if necessary.

Cette thèse porte sur la réalisation des tâches avec la locomotion sur des robots humanoïdes. Grâce à leurs nombreux degrés de liberté, ces robots possèdent un très haut niveau de redondance. D’autre part, les humanoïdes sont sous-actionnés dans le sens où la position et l’orientation ne sont pas directement contrôlées par un moteur. Ces deux aspects, le plus souvent étudiés séparément dans la littérature, sont envisagés ici dans un même cadre. En outre, la génération d’un mouvement complexe impliquant à la fois des tâches de manipulation et de locomotion, étudiée habituellement sous l’angle de la planification de mouvement, est abordée ici dans sa composante réactivité temps réel. En divisant le processus d’optimisation en deux étapes, un contrôleur basé sur la notion de pile de tâches permet l’adaptation temps réel des empreintes de pas planifiées dans la première étape. Un module de perception est également conçu pour créer une boucle fermée de perception-décision-action. Cette architecture combinant planification et réactivité est validée sur le robot HRP-2. Deux classes d’expériences sont menées. Dans un cas, le robot doit saisir un objet éloigné, posé sur une table ou sur le sol. Dans l’autre, le robot doit franchir un obstacle. Dans les deux cas, les condition d’exécution sont mises à jour en temps réel pour faire face à la dynamique de l’environnement : changement de position de l’objet à saisir ou de l’obstacle à franchir. / This thesis focuses on realization of tasks with locomotion on humanoid robots. Thanks to their numerous degrees of freedom, humanoid robots possess a very high level of redundancy. On the other hand, humanoids are underactuated in the sense that the position and orientation of the base are not directly controlled by any motor. These two aspects, usually studied separately in manipulation and locomotion research, are unified in a same framework in this thesis and are resolved as one unique problem. Moreover, the generation of a complex movement involving both tasks and footsteps is also improved becomes reactive. By dividing the optimization process into appropriate stages and by feeding directly the intermediate result to a task-based controller, footsteps can be calculated and adapted in real-time to deal with changes in the environment. A perception module is also developed to build a closed perception-decision-action loop. This architecture combining motion planning and reactivity validated on the HRP-2 robot. Two classes of experiments are carried out. In one case the robot has to grasp an object far away at different height level. In the other, the robot has to step over an object on the floor. In both cases, the execution conditions are updated in real-time to deal with the dynamics of the environment: changes in position of the target to be caught or of the obstacle to be stepped over.

Cette thèse s’inscrit dans le domaine de l’automatique, et a pour but de fournir des outils utiles à l’atterrissage en mer d’hélicoptères (sur navire ou plateforme) et employés dans le cadre d’un potentiel pilotage automatique. L’objectif a donc été de développer une série de lois de commande pilotées manuellement, puis commandées de façon autonome à l’aide d’informations caméra. Les lois ainsi développées à l’aide de modèles dynamiques d’hélicoptères, limitations mécaniques incluses, se basent sur les critères de Qualité de Vol issus de la norme ADS-33. L’ensemble a fait appel à une approche anti-windup pour améliorer la robustesse face aux situations d’actionneurs en saturation. Enfin les lois issues de ces travaux ont été testées en temps-réel sur le banc de pilotage d’hélicoptères de l’ONERA Salon-de-Provence. / This thesis is related to the automatic & control engineering field, and itsmain goal is to provide useful tools for ship landing missions, tools that can be used fora potential autopilot. The objective has been to develop a series of control laws manuallypiloted, then automatically controlled by visual servoing using identified image features. Thelaws developped thanks to helicopter models with mechanical limitations were based on flyingqualities criteria from the ADS-33 standard. The process also defines an anti-windup approachto cope with actuator saturations. Finally the main results were assessed in real time withthe ONERA rotorcraft flight test bench at ONERA Salon-de-Provence.

[pt] Neste trabalho, apresenta-se diferentes esquemas de controle servovisuais para tarefas robóticas de colheita de fruta, na presença de incertezas paramétricas nos modelos do sistema. O primeiro esquema combina as abordagens de servovisão baseada em posição (PBVS) e servovisão baseada em imagem (IBVS) para realizar, respectivamente, a aproximação até a fruta e, em seguida, um ajuste fino para a colheita. O segundo esquema usa uma abordagem de servovisão híbrida (HVS) para realizar a tarefa de colheita completa, projetando uma lei de controle adequada que combina vetores de erro definidos no espaço operacional e no espaço da imagem. A fase de detecção utiliza um algoritmo baseado no espaço de cores OTHA e limiar da imagem Otsu para um rápido reconhecimento de frutos maduros em cenários complexos. Além disso, um método de detecção mais preciso emprega uma Rede Neural Convolucional Profunda (DCNN) pré-treinada baseada em uma versão Segnet minimizada para uma inferência rápida durante a execução da tarefa. A localização do objeto é realizada empregando uma técnica de triangulação de imagem, que combina os algoritmos SURF e RANSAC ou ORB e BF-Matcher para extrair a característica da imagem da fruta e associa-lo com o seu ponto correspondente na outra visualização. No entanto, como esses algoritmos exigem um elevado custo computacional para os requisitos da tarefa, um método de estimativa mais rápido utiliza o centróide da fruta e transformação homogênea para descobrir os pontos correspondentes. Finalmente, um esquema de controle em modos deslizantes (SMC) baseado em visão e uma função de monitoramento de comutação são empregados para lidar com incertezas nos parâmetros de calibração do sistema de câmera-robô. Nesse sentido, é possível garantir a estabilidade assintótica e a convergência do erro da característica da imagem, mesmo que o ângulo de desalinhamento, em torno do eixo z, entre os sistemas de coordenadas da câmera e do efetuador seja incerto. / [en] In this work, we present different eye-in-hand visual servoing control schemes applied to a robotic harvesting task of soft fruits in the presence of parametric uncertainties in the system models. The first scheme combines position-based visual servoing (PBVS) and image-based visual servoing (IBVS) approaches in order to perform respectively an approach phase to the fruit and then a fine tuning of the end-effector to harvest. The second scheme uses a hybrid visual servoing (HVS) approach to fulfill the complete harvesting task, by designing a suitable control law which combines error vectors defined in both the image and operational spaces. For detecting the fruits, an algorithm based on the combination of the OHTA color space and Otsu’s threshold method for a fast recognition of mature fruits in complex scenarios. In addition, a more accurate detection method employs a pre-trained deep encoder-decoder algorithm based on a minimized Segnet version for a fast and cheap inference during the task execution. The object localization is accomplished by employing an image triangulation technique, which combines the speeded-up-robust-features (SURF) and the-randomsample-consensus (RANSAC) or the Oriented FAST and Rotated BRIEF and the Brute-Force Matcher (BF-Matcher) algorithms to extract the fruit image feature and match it to its correspondent feature-point into the other view of the stereo camera. However, since these algorithms are computationally expensive for the task requirements, a faster estimation method uses the fruit centroid and a homogeneous transformation for discovering matching points. Finally, a vision-based sliding-mode-control scheme and a switching monitoring function are employed to cope with uncertainties in the calibration parameters of the camera-robot system. In this context, it is possible to guarantee the asymptotic stability and convergence of the image feature error, even if the misalignment angle, around the z-axis, between the camera and end-effector frames is uncertain. 3D computer simulations and preliminary experimental results, obtained with a Mitsubishi robot arm RV-2AJ carrying out a simple strawberry picking task, are included to illustrate the performance and effectiveness of the proposed control scheme.

Precision landing capability is a necessary development that must take place before unmanned aircraft systems (UAS) can realize their full potential in today's modern society. Current multirotor UAS are heavily reliant on GPS data to provide positioning information for landing. While generally accurate to within several meters, much higher levels of accuracy are needed to ensure safe and trouble-free operations in several UAS applications that are currently being pursued. Examples of these applications include package delivery, automatic docking and recharging, and landing on moving vehicles. The specific problem we consider is that of precision landing of a multirotor unmanned aircraft on a small barge at sea---which presents several significant challenges. Not only must we land on a moving vehicle, but the vessel also experiences random rotational and translational motion as a result of waves and wind. Because maritime operations often span long periods of time, it is also desirable that precision landing can occur at any time---day or night.In this work we present a complete approach for precision shipboard landing and address each of the aforementioned challenges. Our method is enabled by leveraging an on-board camera and a specialized landing target which can be detected in light or dark conditions. Features belonging to the target are extracted from camera imagery and are used to compute image-based visual servoing velocity commands that lead to precise alignment between the multirotor and landing target. To enable the multirotor to match the horizontal velocities of the barge, an extended Kalman filter is used to generate feed-forward velocity reference commands. The complete landing procedure is guided by a state machine architecture that incorporates corrections to account for wind, and is also capable of quickly reacquiring the landing target in a loss event. Our approach is thoroughly validated through full-scale outdoor flight tests and is shown to be reliable, timely, and accurate to within 4 to 10 centimeters.