As imagens são sinais que possuem muita informação. Os objetos representados em ima- gens podem sofrer deformações fazendo com que suas características mudem, o que difi- culta o reconhecimento do objeto. A chave para o sucesso de um sistema de reconheci- mento de padrões em imagens é escolher adequadamente a sua abordagem e um modelo para as feições presentes nas imagens. Uma das dificuldades é extrair e selecionar as feições que são mais discriminantes entre as diferentes classes usadas para modelar um objeto. Os modelos ativos de formas (Active Shape Models ASM) adaptam-se às defor- mações de um objeto. O objeto a ser modelado, pode ser representado com um modelo de pontos distribuídos (Point Distribution Models PDM). O PDM consiste de pontos de interesse ou marcadores, que permitem a extração de feições em localizações específicas do objeto. Após tratar a incerteza da localização e oclusão dos marcadores é possível ex- trair as feições mais representativas, obtendo-se um desempenho alto em termos da taxa de reconhecimento. Nesta tese são introduzidas novas formas para extrair e selecionar feições com modelos ativos de formas, que melhoram a taxa de classificação onde há objetos deformáveis. Esta tese é inovadora no sentido de aperfeiçoar o uso de ASMs na classificação de faces humanas, e na sua aplicação no monitoramento visual de outros tipos de objetos deformáveis. / Images are signals that have a lot of information. The objects depicted in images may suf- fer deformations causing changes in their characteristics, which hinders the recognition of the object. The key to the success of a system of pattern recognition in images is to choose properly your approach and a model for the features in the images. One difficulty is to extract and select the features that are most discriminating between different classes used to model an object. The Active Shape Models (ASM) adapt to deformations of an object. The object to be modeled can be represented with a Points Distribution Model (PDM). The PDM consists of points of interest or landmarks that allow the extraction of features in specific locations of the object. After treating the uncertainty of the location and occlu- sion of the landmarks it is possible to extract the most representative features, obtaining a high performance in terms of recognition rate. This thesis introduces new ways to extract and select features with ASMs, which improve the classification rate where deformable objects are present. This thesis is innovative in the sense that improves the use of ASMs in the classification of human faces, and that can be applied in visual monitoring of other types of deformable objects.

As imagens são sinais que possuem muita informação. Os objetos representados em ima- gens podem sofrer deformações fazendo com que suas características mudem, o que difi- culta o reconhecimento do objeto. A chave para o sucesso de um sistema de reconheci- mento de padrões em imagens é escolher adequadamente a sua abordagem e um modelo para as feições presentes nas imagens. Uma das dificuldades é extrair e selecionar as feições que são mais discriminantes entre as diferentes classes usadas para modelar um objeto. Os modelos ativos de formas (Active Shape Models ASM) adaptam-se às defor- mações de um objeto. O objeto a ser modelado, pode ser representado com um modelo de pontos distribuídos (Point Distribution Models PDM). O PDM consiste de pontos de interesse ou marcadores, que permitem a extração de feições em localizações específicas do objeto. Após tratar a incerteza da localização e oclusão dos marcadores é possível ex- trair as feições mais representativas, obtendo-se um desempenho alto em termos da taxa de reconhecimento. Nesta tese são introduzidas novas formas para extrair e selecionar feições com modelos ativos de formas, que melhoram a taxa de classificação onde há objetos deformáveis. Esta tese é inovadora no sentido de aperfeiçoar o uso de ASMs na classificação de faces humanas, e na sua aplicação no monitoramento visual de outros tipos de objetos deformáveis. / Images are signals that have a lot of information. The objects depicted in images may suf- fer deformations causing changes in their characteristics, which hinders the recognition of the object. The key to the success of a system of pattern recognition in images is to choose properly your approach and a model for the features in the images. One difficulty is to extract and select the features that are most discriminating between different classes used to model an object. The Active Shape Models (ASM) adapt to deformations of an object. The object to be modeled can be represented with a Points Distribution Model (PDM). The PDM consists of points of interest or landmarks that allow the extraction of features in specific locations of the object. After treating the uncertainty of the location and occlu- sion of the landmarks it is possible to extract the most representative features, obtaining a high performance in terms of recognition rate. This thesis introduces new ways to extract and select features with ASMs, which improve the classification rate where deformable objects are present. This thesis is innovative in the sense that improves the use of ASMs in the classification of human faces, and that can be applied in visual monitoring of other types of deformable objects.

As imagens são sinais que possuem muita informação. Os objetos representados em ima- gens podem sofrer deformações fazendo com que suas características mudem, o que difi- culta o reconhecimento do objeto. A chave para o sucesso de um sistema de reconheci- mento de padrões em imagens é escolher adequadamente a sua abordagem e um modelo para as feições presentes nas imagens. Uma das dificuldades é extrair e selecionar as feições que são mais discriminantes entre as diferentes classes usadas para modelar um objeto. Os modelos ativos de formas (Active Shape Models ASM) adaptam-se às defor- mações de um objeto. O objeto a ser modelado, pode ser representado com um modelo de pontos distribuídos (Point Distribution Models PDM). O PDM consiste de pontos de interesse ou marcadores, que permitem a extração de feições em localizações específicas do objeto. Após tratar a incerteza da localização e oclusão dos marcadores é possível ex- trair as feições mais representativas, obtendo-se um desempenho alto em termos da taxa de reconhecimento. Nesta tese são introduzidas novas formas para extrair e selecionar feições com modelos ativos de formas, que melhoram a taxa de classificação onde há objetos deformáveis. Esta tese é inovadora no sentido de aperfeiçoar o uso de ASMs na classificação de faces humanas, e na sua aplicação no monitoramento visual de outros tipos de objetos deformáveis. / Images are signals that have a lot of information. The objects depicted in images may suf- fer deformations causing changes in their characteristics, which hinders the recognition of the object. The key to the success of a system of pattern recognition in images is to choose properly your approach and a model for the features in the images. One difficulty is to extract and select the features that are most discriminating between different classes used to model an object. The Active Shape Models (ASM) adapt to deformations of an object. The object to be modeled can be represented with a Points Distribution Model (PDM). The PDM consists of points of interest or landmarks that allow the extraction of features in specific locations of the object. After treating the uncertainty of the location and occlu- sion of the landmarks it is possible to extract the most representative features, obtaining a high performance in terms of recognition rate. This thesis introduces new ways to extract and select features with ASMs, which improve the classification rate where deformable objects are present. This thesis is innovative in the sense that improves the use of ASMs in the classification of human faces, and that can be applied in visual monitoring of other types of deformable objects.

Les objets déformables sont omniprésents dans notre vie quotidienne. Chaque jour, nous manipulons des vêtements dans des configurations innombrables pour nous habiller, nouons les lacets de nos chaussures, cueillons des fruits et des légumes sans les endommager pour notre consommation et plions les reçus dans nos portefeuilles. Toutes ces tâches impliquent de manipuler des objets déformables et peuvent être exécutées sans problème par une personne. Toutefois, les robots n'ont pas encore atteint le même niveau de dextérité. Contrairement aux objets rigides, que les robots sont maintenant capables de manipuler avec des performances proches de celles des humains; les objets déformables doivent être contrôlés non seulement pour les positionner, mais aussi pour définir leur forme. Cette contrainte supplémentaire, relative au contrôle de la forme d’un objet, rend les techniques utilisées pour les objets rigides inapplicables aux objets déformables. En outre, le comportement des objets déformables diffère largement entre eux, par exemple: la forme d’un câble et des vêtements est considérablement affectée par la gravité, alors que celle-ci n’affecte pas la configuration d’autres objets déformables tels que des produits alimentaires. Ainsi, différentes approches ont été proposées pour des classes spécifiques d’objets déformables.Dans cette thèse, nous cherchons à remédier à ces lacunes en proposant une approche modulaire pour détecter la forme d'un objet pendant qu'il est manipulé par un robot. La modularité de cette approche s’inspire d’un paradigme de programmation qui s’applique de plus en plus au développement de logiciels en robotique et vise à apporter des solutions plus générales en séparant les fonctionnalités en composants. Ces composants peuvent ensuite être interchangés en fonction de la tâche ou de l'objet concerné. Cette stratégie est un moyen modulaire de suivre la forme d'objets déformables.Pour valider la stratégie proposée, nous avons implémenté trois applications différentes. Deux applications portaient exclusivement sur l'estimation de la déformation de l'objet à l'aide de données tactiles ou de données issues d’un capteur d’effort. La troisième application consistait à contrôler la déformation d'un objet. Une évaluation de la stratégie proposée, réalisée sur un ensemble d'objets élastiques pour les trois applications, montre des résultats prometteurs pour une approche qui n'utilise pas d'informations visuelles et qui pourrait donc être améliorée de manière significative par l'ajout de cette modalité. / Deformable objects are ubiquitous in our daily lives. On a given day, we manipulate clothes into uncountable configurations to dress ourselves, tie the shoelaces on our shoes, pick up fruits and vegetables without damaging them for our consumption and fold receipts into our wallets. All these tasks involve manipulating deformable objects and can be performed by an able person without any trouble, however robots have yet to reach the same level of dexterity. Unlike rigid objects, where robots are now capable of handling objects with close to human performance in some tasks; deformable objects must be controlled not only to account for their pose but also their shape. This extra constraint, to control an object's shape, renders techniques used for rigid objects mainly inapplicable to deformable objects. Furthermore, the behavior of deformable objects widely differs among them, e.g. the shape of a cable and clothes are significantly affected by gravity while it might not affect the configuration of other deformable objects such as food products. Thus, different approaches have been designed for specific classes of deformable objects.In this thesis we seek to address these shortcomings by proposing a modular approach to sense the shape of an object while it is manipulated by a robot. The modularity of the approach is inspired by a programming paradigm that has been increasingly been applied to software development in robotics and aims to achieve more general solutions by separating functionalities into components. These components can then be interchanged based on the specific task or object at hand. This provides a modular way to sense the shape of deformable objects.To validate the proposed pipeline, we implemented three different applications. Two applications focused exclusively on estimating the object's deformation using either tactile or force data, and the third application consisted in controlling the deformation of an object. An evaluation of the pipeline, performed on a set of elastic objects for all three applications, shows promising results for an approach that makes no use of visual information and hence, it could greatly be improved by the addition of this modality.

Performing tasks with a robotic hand often requires a complete knowledge of the manipulated object, including its properties (shape, rigidity, surface texture) and its location in the environment, in order to ensure safe and efficient manipulation. While well-established procedures exist for the manipulation of rigid objects, as well as several approaches for the manipulation of linear or planar deformable objects such as ropes or fabric, research addressing the characterization of deformable objects occupying a volume remains relatively limited. The fundamental objectives of this research are to track the deformation of non-rigid objects under robotic hand manipulation using RGB-D data, and to automatically classify deformable objects as either rigid, elastic, plastic, or elasto-plastic, based on the material they are made of, and to support recognition of the category of such objects through a robotic probing process in order to enhance manipulation capabilities. The goal is not to attempt to formally model the material of the object, but rather employ a data-driven approach to make decisions based on the observed properties of the object, capture implicitly its deformation behavior, and support adaptive control of a robotic hand for other research in the future. The proposed approach advantageously combines color image and point cloud processing techniques, and proposes a novel combination of the fast level set method with a log-polar mapping of the visual data to robustly detect and track the contour of a deformable object in a RGB-D data stream. Dynamic time warping is employed to characterize the object properties independently from the varying length of the detected contour as the object deforms. The research results demonstrate that a recognition rate over all categories of material of up to 98.3% is achieved based on the detected contour. When integrated in the control loop of a robotic hand, it can contribute to ensure stable grasp, and safe manipulation capability that will preserve the physical integrity of the object.

Object deformation is an essential factor for the robot to manipulate the object, as the deformation impacts the grasping of the deformable object either positively or negatively. One of the most challenging problems with deformable objects is estimating the stiffness parameters such as Young’s modulus and Poisson’s ratio. This thesis presents a learning-based approach to predicting the stiffness parameters of a 3D (volumetric) deformable object based on vision and haptic feedback. A deep learning network is designed to predict Young’s modulus of homogeneous isotropic deformable objects from the forces of squeezing the object and the depth images of the deformed part of the object. The results show that the developed method can estimate Young’s modulus of the selected synthetic objects in the validation samples dataset with 3.017% error upper bound on the 95% confidence interval. The conclusion is that this method contributes to predicting Young’s modulus of the homogeneous isotropic objects in the simulation environments. In future work, the diversity of the object shape samples can be expanded for broader application in predicting Young’s modulus. Besides, the method can also be extended to real-world objects after validating real-world experiments. / Objekt är en väsentlig faktor för roboten att manipulera objektet, eftersom det påverkar greppet om det deformerbara objektets deformation antingen positivt eller negativt. Ett av de mest utmanande problemen med deformerbara objekt är att uppskatta styvhetsparametrarna som Youngs modul och Poissons förhållande . Denna avhandling presenterar en inlärningsbaserad metod för att förutsäga styvhetsparametrarna för ett 3D (volumetriskt) deformerbart objekt baserat på syn och haptisk feedback. Ett nätverk för djupinlärning är utformat för att förutsäga Youngs modul av homogena isotropa deformerbara objekt från krafterna från att klämma ihop objektet och djupbilderna av den deformerade delen av objektet Resultaten visar att den utvecklade metoden kan uppskatta Youngs modul för de utvalda syntetiska objekten i valideringsexempeldatauppsättningen med 3.017% fel övre gräns på 95% konfidensintervall. Slutsatsen är att denna metod bidrar till att förutsäga Youngs modul för de homogena isotropa objekten i simuleringsmiljöerna. I framtida bredare arbete kan mångfalden av objektformproverna utökas för tillämpning vid förutsägelse av Youngs modul. Dessutom kan metoden också utvidgas till verkliga objekt efter validering av verkliga experiment.

Les techniques d'insertion d'aiguille font partie des interventions chirurgicales les plus courantes. L'efficacité de ces interventions dépend fortement de la précision du positionnement des aiguilles dans un emplacement cible à l'intérieur du corps du patient. L'objectif principal dans cette thèse est de développer un système robotique autonome, capable d'insérer une aiguille flexible dans une structure déformable le long d'une trajectoire prédéfinie. L’originalité de ce travail se trouve dans l’utilisation de simulations inverses par éléments finis (EF) dans la boucle de contrôle du robot pour prédire la déformation des structures. La particularité de ce travail est que pendant l’insertion, les modèles EF sont continuellement recalés (étape corrective) grâce à l’information extraite d’un système d’imagerie peropératoire. Cette étape permet de contrôler l’erreur des modèles par rapport aux structures réelles et ainsi éviter qu'ils divergent. Une seconde étape (étape de prédiction) permet, à partir de la position corrigée, d’anticiper le comportement de structures déformables, en se reposant uniquement sur les prédictions des modèles biomécaniques. Ceci permet ainsi d’anticiper la commande du robot pour compenser les déplacements des tissus avant même le déplacement de l’aiguille. Expérimentalement, nous avions utilisé notre approche pour contrôler un robot réel afin d'insérer une aiguille flexible dans une mousse déformable le long d'une trajectoire (virtuelle) prédéfinie. Nous avons proposé une formulation basée sur des contraintes permettant le calcul d'étapes prédictives dans l'espace de contraintes offrant ainsi un temps d'insertion total compatible avec les applications cliniques. Nous avons également proposé un système de réalité augmentée pour la chirurgie du foie ouverte. La méthode est basée sur un recalage initial semi-automatique et un algorithme de suivi peropératoire basé sur des marqueurs (3D) optiques. Nous avons démontré l'applicabilité de cette approche en salle d'opération lors d'une chirurgie de résection hépatique. Les résultats obtenus au cours de ce travail de thèse ont conduit à trois publications (deux IROS et un ICRA) dans les conférences internationales puis à un journal (Transactions on Robotics) en cours de révision. / Needle-based interventions are among the least invasive surgical approaches to access deep internal structures into organs' volumes without damaging surrounding tissues. Unlike traditional open surgery, needle-based approaches only affect a localized area around the needle, reducing this way the occurrence of traumas and risks of complications \cite{Cowan2011}. Many surgical procedures rely on needles in nowadays clinical routines (biopsies, local anesthesia, blood sampling, prostate brachytherapy, vertebroplasty ...). Radiofrequency ablation (RFA) is an example of percutaneous procedure that uses heat at the tip of a needle to destroy cancer cells. Such alternative treatments may open new solutions for unrespectable tumors or metastasis (concerns about the age of the patient, the extent or localization of the disease). However, contrary to what one may think, needle-based approaches can be an exceedingly complex intervention. Indeed, the effectiveness of the treatment is highly dependent on the accuracy of the needle positioning (about a few millimeters) which can be particularly challenging when needles are manipulated from outside the patient with intra-operative images (X-ray, fluoroscopy or ultrasound ...) offering poor visibility of internal structures. Human factors, organs' deformations, needle deflection and intraoperative imaging modalities limitations can be causes of needle misplacement and rise significantly the technical level necessary to master these surgical acts. The use of surgical robots has revolutionized the way surgeons approach minimally invasive surgery. Robots have the potential to overcome several limitations coming from the human factor: for instance by filtering operator tremors, scaling the motion of the user or adding new degrees of freedom at the tip of instruments. A rapidly growing number of surgical robots has been developed and applied to a large panel of surgical applications \cite{Troccaz2012}. Yet, an important difficulty for needle-based procedures lies in the fact that both soft tissues and needles tend to deform as the insertion proceeds in a way that cannot be described with geometrical approaches. Standard solutions address the problem of the deformation extracting a set of features from per-operative images (also called \textit{visual servoing)} and locally adjust the pose/motion of the robot to compensate for deformations \cite{Hutchinson1996}. [...]To overcome these limitations, we introduce a numerical method allowing performing inverse Finite Element simulations in real-time. We show that it can be used to control an articulated robot while considering deformations of structures during needle insertion. Our approach relies on a forward FE simulation of a needle insertion (involving complex non-linear phenomena such as friction, puncture and needle constraints).[...]