Robotic grasping and manipulation of three-dimensional deformable objects is a complex task that currently does not have robust and flexible solutions. Deformable objects include a wide variety of elastic and inelastic objects that change size and shape during manipulation. The development of adaptable methods for grasping and autonomously controlling the shape of three-dimensional deformable objects will benefit many commercial applications, including shaping parts for assembly in manufacturing, manipulating food for packaging and controlling tissues during robotic surgery. Controlling a deformable object to a desired shape requires first choosing contact points on the object's surface. Next, the robotic hand is positioned in the correct position and orientation to grasp and deform the object. After deformation, the object is assessed to evaluate the quality of the shape control procedure. In many cases, this process is completed without knowing the object's properties or behaviour before deformation. This work proposes and implements the framework for a robotic arm and hand system to control the shape of a previously unseen deformable object autonomously. Significant original contributions are made in developing an original algorithm to plan contact points on a three-dimensional object for grasping and shape control. This research uses a novel object representation to reduce the dimensionality of the deformable object manipulation problem. A path planning algorithm guides the robot arm to the optimal valid grasp pose to deform the object at the determined contact points. Additional contributions include developing a multi-view assessment strategy to determine the quality of the deformation towards the desired shape. The system completes the objectives using depth and colour images captured from a single point of view to locate and identify a previously unseen three-dimensional object within a robotic workspace. After estimating the unknown object's geometry, initial grasp contact points are planned to control the object to the desired shape. The grasp points are used to plan and execute a collision-free trajectory for the robot manipulator to place the robotic hand in the optimal position and orientation to grasp and deform the object. After the deformation is complete, the object is moved to a variety of assessment positions to determine the success of the shape control procedure. The system is validated experimentally on a variety of deformable three-dimensional objects.

Improving robotic manipulation is critical for robots to be actively useful in realworld factories and homes. While some success has been shown in simulation and controlled environments, robots are slow, clumsy, and not general or robust enough when interacting with their environment. By contrast, humans effortlessly manipulate objects. One possible reason for this discrepancy is that, starting from birth, humans have years of experience to collect data and develop good internal models of what happens when they manipulate objects. If robots could also learn models from a large amount of real data, perhaps they, too, could become more capable manipulators. In this thesis, we propose to improve robotic manipulation by solving two problems. First, we look at how robots can collect a large amount of manipulation data without human intervention. Second, we study how to build statistical models of robotic manipulation from the collected data. These data-driven models can then be used for planning more robust manipulation actions. To solve the first problem of enabling large data collection, we perform several different robotic manipulation experiments and use these as case studies. We study bin-picking, post-grasp manipulation, pushing, tray tilting, planar grasping, and regrasping. These case studies allow us to gain insights on how robots can collect a large amount of accurate data with minimal human intervention. To solve the second problem of statistically modeling manipulation actions, we propose models for different parts of various manipulation actions. First, we look at how to model post-grasp manipulation actions by modeling the probability distribution of where an object ends up in a robot’s hand, and how this affects its success rate at various tasks such as placing or insertion. Second, we model how robots can change the pose of an object in their hand with regrasp actions. Third, we improve on the place and pick regrasp action by modeling each separately with more data. These learned data-driven models can then be used for planning more robust and accurate manipulation actions.

We still struggle to deliver autonomous robots that perform manipulation tasks as simple for a human as picking up items. A portion of the difficulty of this task lays on the fact that such operation requires a robot that can deal with uncertainty in an unstructured environment. We propose in this thesis the use of visual and tactile perception for providing solutions that can improve the robustness of a robotic manipulator in such environment. In this thesis, we approach robotic grasping using a single 3D point cloud with a partial view of the objects present in the scene. Moreover, the objects are unknown: they have not been previously recognised and we do not have a 3D model to compute candidate grasping points. In experimentation, we prove that our solution is fast and robust, taking in average 17 ms to find a grasp which is stable 85% of the time. Tactile sensors provide a rich source of information regarding the contact experienced by a robotic hand during the manipulation of an object. In this thesis, we exploit with deep learning this type of data for approaching the prediction of the stability of a grasp and the detection of the direction of slip of a contacted object. We prove that our solutions could correctly predict stability 76% of the time with a single tactile reading. We also demonstrate that learning temporal and spatial patterns leads to detections of the direction of slip which are correct up to 82% of the time and are only delayed 50 ms after the actual slip event begins. Despite the good results achieved on the previous two tactile tasks, this data modality has a serious flaw: it can only be registered during contact. In contrast, humans can estimate the feeling of grasping an object just by looking at it. Inspired by this, we present in this thesis our contributions for learning to generate tactile responses from vision. We propose a supervised solution based on training a deep neural network that models the behaviour of a tactile sensor, given 3D visual information of the target object and grasp data as an input. As a result, our system has to learn to link vision to touch. We prove in experimentation that our system learns to generate tactile responses on a set of 12 items, being off by only 0.06 relative error points. Furthermore, we also experiment with a semi-supervised solution for learning this task with a reduced need of labelled data. In experimentation, we show that it learns our tactile data generation task with 50% less data than the supervised solution, incrementing only 17% the error. Last, we introduce our work in the generation of candidate grasps which are improved through simulation of the tactile responses they would generate. This work unifies the contributions presented in this thesis, as it applies modules on calculating grasps, stability prediction and tactile data generation. In early experimentation, it finds grasps which are more stable than the original ones produced by our method based on 3D point clouds. / This doctoral thesis has been carried out with the support of the Spanish Ministry of Economy, Industry and Competitiveness through the grant BES-2016-078290.

Este trabalho propõe um sistema de manipulação robótica para interação física segura com objetos ou humanos em ambientes não estruturados. A proposta considera a execução de tarefas de manipulação e a prevenção e tratamento de colisões utilizando apenas uma lei de controle, o controle de impedância. A inovação científica consiste em um sistema multifuncional implementado com uma única lei de controle em contraste com os sistemas já existem que utilizam chaveamento entre controladores para cada diferente funcionalidade do sistema, e que apresentam diversas desvantagens como instabilidade e oscilações, aumento da complexidade de programação, entre outras. Inicialmente é proposto um planejador de manipulação e regrasping baseado na combinação de trajetórias suaves e na adaptação dos parâmetros de um controle de impedância em tempo de execução. A mudança da impedância para cada etapa é obtida pela modificação dos parâmetros de inércia, rigidez e amortecimento do controlador. A estabilidade desta mudança dinâmica é possível pela utilização de trajetórias suaves obtidas com planejador Squeezed Screw modificado, cujas trajetórias geradas são livres de descontinuidades na posição e na velocidade. Adicionalmente, a prevenção de colisões é realizada com o auxílio de campos potenciais de forças de repulsão formados pela análise de dados de um sistema de visão também proposto. Estes mesmos dados são utilizados para a construção de um mapa de impedâncias ao redor do objeto cuja finalidade é suavizar efeitos de colisões indesejadas. Experimentos com um robô de arquitetura aberta e com um sistema de visão de baixo custo foram realizados na execução tarefa de manipulação de referência para se avaliar o desempenho da metodologia proposta em diferentes condições de operação encontradas em ambientes não estruturados, como por exemplo: erros de medida de posição, de calibração, ocorrência de colisões, etc. A tarefa de manipulação eleita foi a reorientação em 60° de um objeto circular no plano. Os resultados obtidos nos experimentos mostram a capacidade do controle de impedância associado a trajetórias suaves de realizar a tarefa eleita segundo avaliação utilizando como métricas de desempenho a porcentagem de reorientação, que apresentou uma média de 80% mesmo na presença de erros de medida do sensor de visão e erros de determinação da posição do objeto. / Recent applications in various robotics areas consider interaction between robots and objects or humans in non-structured environments. Under these conditions, in addition to the desire of robots to be able to perform their main tasks, handling, navigation, rehabilitation, etc, it is also desired to prevent and properly handle possible unwanted collisions, whether with objects, with other robots, animals or humans. There are several proposed methods for avoidance, handling and reaction for collisions, however, a widely used strategy is the controller switching between different robot states. There are several drawbacks within this strategy: instability and oscillation, increased programming complexity and consequent increased failure risk, need for different sensors and consequent increase in cost, among others. This work proposes a system applied to the robotic manipulation which is based on only one control law, the impedance control, whose expected capacity is, further performing manipulation tasks, avoidance and handling of potential undesired collisions. It is initially proposed a manipulation planner based the combination of smooth trajectories and the adjustment of parameters an impedance control at runtime. The change of impedance for each phase is achieved by modifying the parameters: mass, spring and damping controller. The stability of this dynamic change is possible by using smooth trajectories obtained with a modified Squeezed Screw trajectory planner, whose paths are discontinuities free in the position and speed. Additionally, collision avoidance is achieved through potential fields the repulsive forces of formed by analysis of data vision. The same data is used to construct an impedance map surrounding the object which objective is collision handling. Experiments with an open architecture robot and a low cost vision system are carried out in the execution of a benchmark manipulation task to evaluate the proposal performance under different operating conditions found in unstructured environments, for example, position measurement errors, calibration problems, occurrence of collisions, among others.

"A number of outstanding problems in robotic motion and manipulation involve tasks where degrees of freedom (DoF), be they part of the robot, an object being manipulated, or the surrounding environment, cannot be accurately controlled by the actuators of the robot alone. Rather, they are also controlled by physical properties or interactions - contact, robot dynamics, actuator behavior - that are influenced by the actuators of the robot. In particular, we focus on two important areas of poorly controlled robotic manipulation: motion planning for deformable objects and in deformable environments; and manipulation with uncertainty. Many everyday tasks we wish robots to perform, such as cooking and cleaning, require the robot to manipulate deformable objects. The limitations of real robotic actuators and sensors result in uncertainty that we must address to reliably perform fine manipulation. Notably, both areas share a common principle: contact, which is usually prohibited in motion planners, is not only sometimes unavoidable, but often necessary to accurately complete the task at hand. We make four contributions that enable robot manipulation in these poorly controlled tasks: First, an efficient discretized representation of elastic deformable objects and cost function that assess a ``cost of deformation' for a specific configuration of a deformable object that enables deformable object manipulation tasks to be performed without physical simulation. Second, a method using active learning and inverse-optimal control to build these discretized representations from expert demonstrations. Third, a motion planner and policy-based execution approach to manipulation with uncertainty which incorporates contact with the environment and compliance of the robot to generate motion policies which are then adapted during execution to reflect actual robot behavior. Fourth, work towards the development of an efficient path quality metric for paths executed with actuation uncertainty that can be used inside a motion planner or trajectory optimizer."

Manipulação robótica em ambientes não estruturados desafia intensamente a comunidade científica. Grandes esforços tem sido empregados em sistemas de visão computacional culminando em resultados notáveis. Entretanto, a informação obtida de sistemas de visão diversas vezes é incompleta, ruidosa e imprecisa devido a limitações técnicas dos sensores empregados ou devido a posição em que o sensor se encontra em relação ao ambiente. Além disso, pouca atenção tem sido destinada a utilização do tato como informação adicional para enriquecer a percepção do ambiente. Este trabalho apresenta um estudo experimental de exploração do ambiente através de contato estável utilizando controle de impedância aplicado a um manipulador robótico. Um framework de manipulação robótica é apresentado e o trabalho é posicionado neste framework na forma de sub-tarefas do mesmo. A descrição do sistema robótico utilizado assim como o detalhamento da implementação de comunicação e controle dos diversos sub-sistemas que o compõem são apresentadas. O trabalho descreve uma série de ensaios experimentais onde o manipulador executa o contato físico com o ambiente. Nestes ensaios é realizada a modificação dos parâmetros do controlador de impedância do manipulador com o objetivo de analisar a influência dos mesmos no resultado. A análise é feita com relação a estabilidade durante o contato e ao erro de acompanhamento de posição após a perda de contato. Em seguida, uma série de estratégias de alteração dos parâmetros do controlador de impedância durante a execução da tarefas é apresentada juntamente com os resultados experimentais e análises comparativas entre elas e os resultados anteriores. O objetivo destas estratégias é melhorar o desempenho e a estabilidade da tarefa garantindo uma medida precisa e estável da posição do objeto no ambiente. A análise dos resultados obtidos mostra que as estratégias efetivamente melhoram o desempenho do sistema em relação aos parâmetros de análise adotados. Finalmente são apresentados os resultados de um teste de repetibilidade da medição da posição de conjunto através da abordagem apresentada. Estes resultados são comparados com resultados de precisão de sistemas de visão computacional. Este teste suporta a conclusão que confirma a viabilidade do método explorado no trabalho. / Robotic manipulation in unstructured environments is a major challenge to the research community. Great efforts are being directed to computational vision systems development and incredible outcome has been achieved. However, the information retrieved from vision systems is often incomplete, noisy or inaccurate due to technical limitations linked to the sensors used in such systems or the positioning of the system in the environment. Furthermore, few attention is been delivered to the application of tactile information to increase the information quality about the environment. This work presents an experimental study about the environment exploration using stable contact and impedance controlled manipulators. A framework for robotic manipulation is presented and this work is positioned in such framework in a tasks and subtasks style. The detailed information about the manipulator system and the implementation is also outlined including all the control levels and communication layers is also outlined. The work describes a series of experimental tests where the manipulator performs physical contact to the environment. The impedance control parameters are than changed aiming to analyse and determine their influence into the observed results. The contact stability and the following error are used as performance indicators. Following such experimental series, four impedance control parameter change strategies are proposed, tested and analysed when performing a task of touching the environment. The results are also compared to the results obtained from the fix parameter tests. The strategies objective is improve the contact stability ensuring a accurate measurement of the environment position. The results show a real improvement of the environment position measurement towards the same measurement when using fixed impedance control parameters. Finally, results from a repeatability test for the results of environment position measurement using the best approach proposed where presented. Such results are compared from the results achieved from vision systems and show a greater performance for the tactile environment exploration approach.

En aplicaciones de captura robotizada se ha hecho necesario el uso de información tridimensional de los objetos que son manipulados. Esta información puede obtenerse mediante dispositivos de adquisición 3D, tales como escáneres láser o cámaras de tiempo de vuelo, que proporcionan imágenes de rango de los objetos. En este trabajo de tesis se presenta un nuevo enfoque para encontrar, sin disponer de una estimación previa, la transformación rígida que produzca una alineación adecuada de las nubes de puntos obtenidas con esos dispositivos. El algoritmo realiza una búsqueda iterativa de correspondencias mediante la comparación de descriptores 2D en varios niveles de resolución utilizando para ello una medida de similitud específicamente diseñada para el descriptor propuesto en esta tesis. Este algoritmo de alineamiento se puede utilizar tanto para modelado 3D como para aplicaciones de manipulación de objetos en situaciones en las que los objetos estén parcialmente ocluidos o presenten simetrías. / In applications of robotic manipulation of objects, the use of three-dimensional information of objects being manipulated has been made necessary. This information can be obtained by 3D acquisition devices, such as laser scanners or cameras of flight time, providing range images of objects. This thesis presents a new approach to find, without having a previous estimate, the Euclidean transformation that produces a proper alignment of point clouds obtained with these devices. The algorithm performs an iterative search for correspondences by comparing 2D descriptors at various levels of resolution using a similarity measure specifically designed for the descriptor proposed in this thesis. This alignment algorithm can be used for both 3D modelling and robotic manipulation applications when objects are partially occluded or have symmetries.

When a robotic system executes a task, there are a number of responsibilities that belong to either the operator and/or the robot. A more autonomous system has more responsibilities in the completion of a task and must possess the decision making skills necessary to adequately deal with these responsibilities. The system must also handle environmental constraints that limit the region of operability and complicate the execution of tasks. There are decisions about the robot’s internal configuration and how the manipulator should move through space, avoid obstacles, and grasp objects. These motions usually have limits and performance requirements associated with them. Successful completion of tasks in a given environment is aided by knowledge of the robot’s capabilities in its workspace. This not only indicates if a task is possible but can suggest how a task should be completed. In this work, we develop a grasping strategy for selecting and attaining grasp configurations for flexible tasks in environments containing obstacles. This is done by sampling for valid grasping configurations at locations throughout the workspace to generate a task plane. Locations in the task plane that contain more valid configurations are stipulated to have higher dexterity and thus provide greater manipulability of targets. For valid configurations found in the plane, we develop a strategy for selecting which configurations to choose when grasping and/or placing an object at a given location in the workspace. These workspace task planes can also be utilized as a design tool to configure the system around the manipulator’s capabilities. We determine the quality of manipulator positioning in the workspace based on manipulability and locate the best location of targets for manipulation. The knowledge of valid manipulator configurations throughout the workspace can be used to extend the application of task planes to motion planning between grasping configurations. This guides the end-effector through more dexterous workspace regions and to configurations that move the arm away from obstacles. The task plane technique employed here accurately captures a manipulator’s capabilities. Initial tests for exploiting these capabilities for system design and operation were successful, thus demonstrating this method as a viable starting point for incrementally increasing system autonomy. / text

Este trabalho propõe um sistema de manipulação robótica para interação física segura com objetos ou humanos em ambientes não estruturados. A proposta considera a execução de tarefas de manipulação e a prevenção e tratamento de colisões utilizando apenas uma lei de controle, o controle de impedância. A inovação científica consiste em um sistema multifuncional implementado com uma única lei de controle em contraste com os sistemas já existem que utilizam chaveamento entre controladores para cada diferente funcionalidade do sistema, e que apresentam diversas desvantagens como instabilidade e oscilações, aumento da complexidade de programação, entre outras. Inicialmente é proposto um planejador de manipulação e regrasping baseado na combinação de trajetórias suaves e na adaptação dos parâmetros de um controle de impedância em tempo de execução. A mudança da impedância para cada etapa é obtida pela modificação dos parâmetros de inércia, rigidez e amortecimento do controlador. A estabilidade desta mudança dinâmica é possível pela utilização de trajetórias suaves obtidas com planejador Squeezed Screw modificado, cujas trajetórias geradas são livres de descontinuidades na posição e na velocidade. Adicionalmente, a prevenção de colisões é realizada com o auxílio de campos potenciais de forças de repulsão formados pela análise de dados de um sistema de visão também proposto. Estes mesmos dados são utilizados para a construção de um mapa de impedâncias ao redor do objeto cuja finalidade é suavizar efeitos de colisões indesejadas. Experimentos com um robô de arquitetura aberta e com um sistema de visão de baixo custo foram realizados na execução tarefa de manipulação de referência para se avaliar o desempenho da metodologia proposta em diferentes condições de operação encontradas em ambientes não estruturados, como por exemplo: erros de medida de posição, de calibração, ocorrência de colisões, etc. A tarefa de manipulação eleita foi a reorientação em 60° de um objeto circular no plano. Os resultados obtidos nos experimentos mostram a capacidade do controle de impedância associado a trajetórias suaves de realizar a tarefa eleita segundo avaliação utilizando como métricas de desempenho a porcentagem de reorientação, que apresentou uma média de 80% mesmo na presença de erros de medida do sensor de visão e erros de determinação da posição do objeto. / Recent applications in various robotics areas consider interaction between robots and objects or humans in non-structured environments. Under these conditions, in addition to the desire of robots to be able to perform their main tasks, handling, navigation, rehabilitation, etc, it is also desired to prevent and properly handle possible unwanted collisions, whether with objects, with other robots, animals or humans. There are several proposed methods for avoidance, handling and reaction for collisions, however, a widely used strategy is the controller switching between different robot states. There are several drawbacks within this strategy: instability and oscillation, increased programming complexity and consequent increased failure risk, need for different sensors and consequent increase in cost, among others. This work proposes a system applied to the robotic manipulation which is based on only one control law, the impedance control, whose expected capacity is, further performing manipulation tasks, avoidance and handling of potential undesired collisions. It is initially proposed a manipulation planner based the combination of smooth trajectories and the adjustment of parameters an impedance control at runtime. The change of impedance for each phase is achieved by modifying the parameters: mass, spring and damping controller. The stability of this dynamic change is possible by using smooth trajectories obtained with a modified Squeezed Screw trajectory planner, whose paths are discontinuities free in the position and speed. Additionally, collision avoidance is achieved through potential fields the repulsive forces of formed by analysis of data vision. The same data is used to construct an impedance map surrounding the object which objective is collision handling. Experiments with an open architecture robot and a low cost vision system are carried out in the execution of a benchmark manipulation task to evaluate the proposal performance under different operating conditions found in unstructured environments, for example, position measurement errors, calibration problems, occurrence of collisions, among others.

Manipulação robótica em ambientes não estruturados desafia intensamente a comunidade científica. Grandes esforços tem sido empregados em sistemas de visão computacional culminando em resultados notáveis. Entretanto, a informação obtida de sistemas de visão diversas vezes é incompleta, ruidosa e imprecisa devido a limitações técnicas dos sensores empregados ou devido a posição em que o sensor se encontra em relação ao ambiente. Além disso, pouca atenção tem sido destinada a utilização do tato como informação adicional para enriquecer a percepção do ambiente. Este trabalho apresenta um estudo experimental de exploração do ambiente através de contato estável utilizando controle de impedância aplicado a um manipulador robótico. Um framework de manipulação robótica é apresentado e o trabalho é posicionado neste framework na forma de sub-tarefas do mesmo. A descrição do sistema robótico utilizado assim como o detalhamento da implementação de comunicação e controle dos diversos sub-sistemas que o compõem são apresentadas. O trabalho descreve uma série de ensaios experimentais onde o manipulador executa o contato físico com o ambiente. Nestes ensaios é realizada a modificação dos parâmetros do controlador de impedância do manipulador com o objetivo de analisar a influência dos mesmos no resultado. A análise é feita com relação a estabilidade durante o contato e ao erro de acompanhamento de posição após a perda de contato. Em seguida, uma série de estratégias de alteração dos parâmetros do controlador de impedância durante a execução da tarefas é apresentada juntamente com os resultados experimentais e análises comparativas entre elas e os resultados anteriores. O objetivo destas estratégias é melhorar o desempenho e a estabilidade da tarefa garantindo uma medida precisa e estável da posição do objeto no ambiente. A análise dos resultados obtidos mostra que as estratégias efetivamente melhoram o desempenho do sistema em relação aos parâmetros de análise adotados. Finalmente são apresentados os resultados de um teste de repetibilidade da medição da posição de conjunto através da abordagem apresentada. Estes resultados são comparados com resultados de precisão de sistemas de visão computacional. Este teste suporta a conclusão que confirma a viabilidade do método explorado no trabalho. / Robotic manipulation in unstructured environments is a major challenge to the research community. Great efforts are being directed to computational vision systems development and incredible outcome has been achieved. However, the information retrieved from vision systems is often incomplete, noisy or inaccurate due to technical limitations linked to the sensors used in such systems or the positioning of the system in the environment. Furthermore, few attention is been delivered to the application of tactile information to increase the information quality about the environment. This work presents an experimental study about the environment exploration using stable contact and impedance controlled manipulators. A framework for robotic manipulation is presented and this work is positioned in such framework in a tasks and subtasks style. The detailed information about the manipulator system and the implementation is also outlined including all the control levels and communication layers is also outlined. The work describes a series of experimental tests where the manipulator performs physical contact to the environment. The impedance control parameters are than changed aiming to analyse and determine their influence into the observed results. The contact stability and the following error are used as performance indicators. Following such experimental series, four impedance control parameter change strategies are proposed, tested and analysed when performing a task of touching the environment. The results are also compared to the results obtained from the fix parameter tests. The strategies objective is improve the contact stability ensuring a accurate measurement of the environment position. The results show a real improvement of the environment position measurement towards the same measurement when using fixed impedance control parameters. Finally, results from a repeatability test for the results of environment position measurement using the best approach proposed where presented. Such results are compared from the results achieved from vision systems and show a greater performance for the tactile environment exploration approach.