This thesis focuses on the definition of a task for the determination, tracking and execution of a grasp on an unknown object. In particular, it is considered the case in which the object is ideally planar and the grasp has to be executed with a two-fingered, parallel-jaw gripper using vision as the source of sensor data. For the specification of this task, an architecture is defined that is based on three basic components -virtual sensors, filters, and actuators-, which can be connected to define a control loop. Each step in this task is analyzed separately, considering several options in some cases.Some of the main contributions of this thesis include: (1) the use of a modular approach to the specification of a control task that provides a basic framework for supporting the concept of behavior; (2) the analysis of several strategies for obtaining a compact representation of the contour of an object; (3) the development of a method for the evaluation and search of a grasp on a planar object for a two-fingered gripper; (4) the specification of different representations of a grasp and the analysis of their use for tracking the grasp between different views of an object; (5) the specification of algorithms for the tracking of a grasp along the views of an object obtained from a sequence of single images and a sequence of stereo images; (6) the definition of parametrized models of the target position of the grasp points and of the feasibility of this target grasp, and of an off-line procedure for the computation of some of the reference values required by this model; and (7) the definition and analysis of a visual servoing control scheme to guide the gripper of a robot arm towards an unknown object using the grasp points computed for that object as control features.

The field of autonomous vehicle navigation and localization is a highly active research topic. The aim of this thesis is to evaluate the feasibility to use outdoor visual navigation in a semi-structured environment. The goal is to develop a visual navigation system for an autonomous golf ball collection vehicle operating on driving ranges. The image feature extractors SIFT and PCA-SIFT was evaluated on an image database consisting of images acquired from 19 outdoor locations over a period of several weeks to allow different environmental conditions. The results from these tests show that SIFT-type feature extractors are able to find and match image features with high accuracy. The results also show that this can be improved further by a combination of a lower nearest neighbour threshold and an outlier rejection method to allow more matches and a higher ratio of correct matches. Outliers were found and rejected by fitting the data to a homography model with the RANSAC robust estimator algorithm. A simulator was developed to evaluate the suggested system with respect to pixel noise from illumination changes, weather and feature position accuracy as well as the distance to features, path shapes and the visual servoing target image (milestone) interval. The system was evaluated on a total of 3 paths, 40 test combinations and 137km driven. The results show that with the relatively simple visual servoing navigation system it is possible to use mono-vision as a sole sensor and navigate semi-structured outdoor environments such as driving ranges.

The wheelchair-mounted robotic arm (WMRA) is a 9-degree of freedom (DoF) assistive system that consists of a 2-DoF modified commercial power wheelchair and a custom 7-DoF robotic arm. Kinematics and control methodology for the 9-DoF system that combine mobility and manipulation have been previously developed and implemented. This combined control allows the wheelchair and robotic arm to follow a single trajectory based on weighted optimizations. However, for the execution of activities of daily living (ADL) in the real-world environment, modified control techniques have been implemented. In order to execute macro ADL tasks, such as a "go to and pick up" task, this work has implemented several control algorithms on the WMRA system. Visual servoing based on template matching and feature extraction allows the mobile platform to approach the desired goal object. Feature extraction based on scale-invariant feature transform (SIFT) gives the system object detection capabilities to recommend actions to the user and to orient the arm to grasp the goal object using visual servoing. Finally, a collision avoidance system is implemented to detect and avoid obstacles when the wheelchair platform is moving towards the goal object. These implementations allow the WMRA system to operate autonomously from the beginning of the task where the user selects the goal object, all the way to the end of the task where the task has been fully completed.

Η παρούσα διδακτορική διατριβή έχει ως αντικείμενο την ανάπτυξη στρατηγικών ελέγχου του ρομπότ για το χειρισμό υφασμάτων κατά τη διαδικασία της ραφής. Ένα βασικό πρόβλημα που παρουσιάζουν τα υφάσματα σε σχέση με τα στερεά αντικείμενα είναι η δυσκολία χειρισμού τους εξαιτίας της πολύ δύσκολα προβλέψιμης συμπεριφοράς τους. Πιο συγκεκριμένα, τα υφάσματα έχουν χαμηλή αντίσταση σε κάμψη που συνεπάγεται την εμφάνιση παραμορφώσεων που μεταβάλλουν το σχήμα τους και επιπλέον, παρουσιάζουν έντονη μη-γραμμικότητα και ανισοτροπία με αποτέλεσμα τη μεγάλη δυσκολία μοντελοποίησής τους κυρίως για εφαρμογές πραγματικού χρόνου. Η έρευνα για την εκπόνηση της παρούσας διδακτορικής διατριβής επικεντρώθηκε στην ανάπτυξη στρατηγικών ελέγχου που βασίζονται σε μεθόδους Υπολογιστικής Νοημοσύνης (Ασαφή Λογική, Γενετικούς Αλγόριθμους και Νευρωνικά Δίκτυα) και Ανάδραση Όρασης. Τα ευφυή συστήματα ελέγχου με χρήση τεχνητής όρασης παρέχουν τη δυνατότητα στο ρομπότ να διεκπεραιώσει με επιδεξιότητα εργασίες σχετικές με τη ραφή υφασμάτων σε πραγματικό περιβάλλον με σκοπό την υψηλότερη ευελιξία και αυτοματισμό. Οι ειδικότεροι στόχοι της διατριβής είναι η ελαχιστοποίηση του συνολικού χρόνου εργασίας του ρομπότ για την ολοκλήρωση της ραφής και ο περιορισμός των σφαλμάτων στη ραφή μέσα στα αποδεκτά όρια. Στο πλαίσιο της διατριβής αυτής, αναπτύχθηκε ένα ευφυές σύστημα για τη ραφή υφασμάτων που υλοποιήθηκε σε μια εργαστηριακή διάταξη που περιλαμβάνει ρομπότ, κάμερες και ραπτομηχανή, καθώς και μια ποικιλία υφασμάτων που χρησιμοποιήθηκαν ως δοκίμια. Το ρομποτικό σύστημα ραφής υφασμάτων περιλαμβάνει διαδικασίες που προηγούνται της ραφής, καθώς και τη διαδικασία της ραφής πάνω στο τραπέζι εργασίας μέχρι να ολοκληρωθούν όλες οι ραφές στο ύφασμα. Για τη ραφή των υφασμάτων πάνω στο τραπέζι εργασίας, αναπτύχθηκε ένα Ασαφές Σύστημα ελέγχου χρησιμοποιώντας μια ποικιλία διαφορετικών υφασμάτων. Ιδιαίτερη έμφαση δόθηκε στην ανάπτυξη του συστήματος ικανού να ανταπεξέρχεται στις πτυχώσεις που εμφανίζονται κατά τη διάρκεια του χειρισμού του υφάσματος από το ρομπότ, χωρίς να οδηγείται σε αστοχία. Στη συνέχεια, αναπτύχθηκαν μέθοδοι βελτιστοποίησης της διαδικασίας της ραφής με κριτήριο την ελαχιστοποίηση του συνολικού χρόνου εργασίας της ραφής θέτοντας εκ των προτέρων τα μέγιστα αποδεκτά σφάλματα για τη ραφή. Η ρύθμιση των παραμέτρων του Ασαφούς Συστήματος πραγματοποιήθηκε με Γενετικούς Αλγόριθμους εκτός της παραγωγικής διαδικασίας (off-line) και με ένα εποπτεύον Ασαφές Σύστημα εντός της παραγωγικής διαδικασίας (on-line). Για την πειραματική επαλήθευση του συστήματος, χρησιμοποιήθηκαν υφάσματα που αποτελούνται από ευθύγραμμα τμήματα. Σε επόμενο στάδιο, αναπτύχθηκε μια στρατηγική ελέγχου για τη ραφή υφασμάτων που αποτελούνται από καμπύλα τμήματα με αυθαίρετη καμπυλότητα. Η προτεινόμενη μέθοδος συνδυάζει τη μέθοδο εντοπισμού των κυρίαρχων σημείων με ένα μικρο-Γενετικό Αλγόριθμο με σκοπό την πολυγωνική προσέγγιση των καμπύλων τμημάτων. Το πρόβλημα που διαμορφώθηκε είχε στόχο την ελαχιστοποίηση των πολυγωνικών τμημάτων που προσεγγίζουν τα καμπύλα τμήματα με δεδομένα τα μέγιστα ανεκτά όρια για το σφάλματα στη ραφή. Επιπλέον, αναπτύχθηκε ένα προσαρμοστικό νευρο-ασαφές σύστημα για τη ραφή υφασμάτων με καμπύλα τμήματα που έχει την ικανότητα να μαθαίνει από πρωθύστερη γνώση. Το σύστημα εκπαιδεύτηκε διεξάγοντας πειράματα ραφής με ένα αριθμό υφασμάτων διαφορετικών καμπυλοτήτων, οπότε καθίσταται ικανό να ανταποκριθεί με αξιοπιστία στη ραφή άλλων υφασμάτων, δηλ. υφασμάτων που δε χρησιμοποιήθηκαν στη διαδικασία εκπαίδευσης. Η δημιουργία του νευρο-ασαφούς αυτού συστήματος βασίστηκε στη χρήση μιας πρωτότυπης μεθόδου ομαδοποίησης δεδομένων. Η προτεινόμενη αυτή μέθοδος ομαδοποίησης βασίστηκε στην ανάπτυξη ενός Γενετικού Αλγόριθμου με χρωμοσώματα μεταβλητού μήκους, ο οποίος έχει το πλεονέκτημα ότι εξασφαλίζει ευελιξία όσον αφορά στον αριθμό των ομάδων που προκύπτουν. Η συμβολή της μεθόδου είναι διττή: αφενός, παράγει αυτόματα τον αριθμό των κέντρων των ομάδων και αφετέρου, αναζητά τα κέντρα σε όλο το πεδίο ορισμού, χωρίς να περιορίζεται στα δεδομένα. Η προτεινόμενη μέθοδος έχει γενικότερη αξία και δεν περιορίζεται μόνο στη χρήση της από το νευρο-ασαφές σύστημα. Αξίζει να σημειωθεί ότι όλες οι στρατηγικές ελέγχου του ρομπότ που αναπτύχθηκαν για το χειρισμό των υφασμάτων, εφαρμόστηκαν σε υφάσματα διαφορετικά ως προς το σχήμα (με ευθύγραμμα και καμπύλα τμήματα), το χρώμα και τις ιδιότητες / The objective of this thesis is the development of control strategies for robot handling of flexible sheets towards the sewing. Besides the difficulties that emerge when handling rigid materials using robots, flexible materials pose additional problems due to due to their unpredictable behavior. In particular, fabrics present low resistance in bending that leads to the appearance of deformations that change their shape and present non-linearity and anisotropy, which poses difficulty in modeling them for real-time applications. The research for this thesis has been focused on the development of control strategies based on Artificial Intelligence techniques (Fuzzy Logic, Genetic algorithms and Neural Networks) and Visual Servoing. The intelligent control systems with artificial vision enable robot to perform skilful tasks related to sewing fabrics in realistic environments towards higher flexibility and automation. The control strategies that have been developed are based on Artificial Intelligence techniques (Fuzzy Logic, Genetic algorithms and Neural Networks) and Visual Servoing. The basic goals of this thesis are the minimization of the total time for robot sewing fabrics and the constraint for the stitching errors inside the acceptable limits. In the context of this thesis, a complete intelligent system has been developed for the handling of fabrics towards sewing. This system is comprised of a robot, two cameras and a sewing machine and a wide range of fabric pieces that was used for experimental purposes. The sewing process is decomposed into preprocess planning and on-line handling subtasks (transferring towards the needle, stitching process and the rotation around the needle). A fuzzy control system was developed for robot handling fabrics on a worktable using a wide range of fabrics. Special emphasis was also given on the development of a system capable of tolerating deformations that may appear on the fabric towards robot handling. Next, optimization methods concerning the handling subtasks were developed in the direction of minimizing the total time for robot sewing fabrics considering the maximum allowable error limits. The parameters were tuned using Genetic Algorithms as an off-line process and a Supervisory fuzzy system in an on-line process. Fabrics comprised of straight edges were used for the experimental verification of the system. The next step concerns the development of a control strategy for robot sewing fabric comprised of curved edges with arbitrary curvatures. The proposed method combines the dominant point detection approach with a micro-Genetic Algorithm for the polygonal approximation of the curved edges. The optimization problem aims at the minimization of the polygonal edges that approximate the curved edges without exceeding the maximum acceptable error limits. In addition, an adaptive neuro-fuzzy system for robot sewing fabrics of curved edges is developed, which has learning capabilities. The system was trained through experiments with various fabrics of different curvatures and is capable to respond to new fabrics, which had not been included in the training process. The construction of the proposed neuro-fuzzy system is based on the use of a novel clustering method. The proposed clustering method is based on the development of a Genetic Algorithm with variable-length chromosomes that has the advantage of flexibility as far as the number of the resulting clusters is concerned. The contribution of the proposed method is twofold. On the one hand, the method evolves automatically the appropriate number of cluster centers, as well as the partitioning of the data, without a priori assumption on the cluster centers. On the other hand, it searches for candidate cluster centers in the universe of discourse and not only among data. The proposed approach is general and it is not limited to the construction of the neuro-fuzzy system. It is also worth noting that all developed control strategies have been applied to fabrics of different shape (with or without curvatures), color and properties.

No description available.

Closing the control loop of a manipulator robot with vision feedback is widelyknown. It concerns nowadays all areas of robotics. Such a return can make a comparison between a desired state and current state, using visual measurements. The main objective of this doctoral thesis is to design several types of kinematic control laws for stereo visual servoing. It strongly involves the formalism of the task function which is a well-known and useful mathematical tool to express the visual error as a function of state vectors.We have investigated the decoupling between the rotational and translational velocities control laws together with the epipolar constraint with a stereo visual feedback.That is why, the visual measurements and features used in this thesis are the 3Dstraight lines.The interests of this type of visual features rely on the robustness against the noise, and the possibility to represent straight lines or other features like points or planes pairs by the Plücker coordinates, as a 3D straight line can be represented as well by two points or the intersection of two planes. This makes all the control laws designed in this thesis valid for another visual features like points

La vision artificielle est un moyen de perception très apprécié en robotique. Les applications les plus courantes de la vision en robotique manipulatrice sont l'estimation de pose et la commande. D'un point de vue conceptuel, l'utilisation de la vision artificielle permet d'améliorer les performances de la commande des robots en termes de précision et de robustesse (vis-à-vis des erreurs sur les paramètres géométriques du robot). De plus, cette mesure est d'autant plus pertinente pour les robots parallèles puisque l'état de ces derniers est généralement mieux défini par la pose de l'effecteur que par les mesures articulaires, traditionnellement utilisées en commande. Cependant, les systèmes de vision classiques ne permettent pas de satisfaire les exigences des commandes hautes performances à cause de leur période d'acquisition et de leur temps de latence trop grands. Pour pallier ce problème, l'approche proposée dans cette thèse est de procéder à une acquisition séquentielle de fenêtres d'intérêt. En effet, le fait de ne transmettre que les régions de l'image contenant les primitives visuelles utiles a pour effet de diminuer la quantité de données à transmettre ce qui permet réduire la période d'acquisition et le temps de latence. De plus, l'acquisition non simultanée des primitives offre la possibilité d'estimer la pose et la vitesse du robot de façon conjointe. Différentes méthodes d'estimation et plusieurs schémas de commandes cinématiques et dynamiques utilisant ce mode d'acquisition ont ainsi été proposés dans ce document. Les résultats expérimentaux obtenus en commande dynamique par vision d'un robot parallèle montrent, pour la première fois, que la commande référencée vision peut être plus performante que la commande articulaire. / Visual sensing is very appreciated in manipulation robotics since it provides measures for pose estimation and robot control. Conceptually, vision allows improving the performance of manipulator robots control from accuracy and robustness (against kinematic parameters errors) points of view. In addition, vision-based control is particularly relevant for parallel robot manipulators. In fact, the state of these robots is usually better described by the pose of their mobile platform than by their articular joints. Thus, the use of vision simplifies the control of parallel robots. However, the typical vision systems do not fulfil the dynamic control constraints on the acquisition frequency and the latency. The approach proposed in this thesis is to perform a sequential acquisition of regions of interest which contain the useful visual features to cut down the data amount to transmit. Thus allows for reducing the acquisition period and latency. In addition, the sequential acquisition allows for estimating both the pose and the velocity of the robot platform. Thanks to this acquisition method, several control laws are proposed. The experimental results show that the performance of the proposed vision based dynamic control laws are, for the first time, better than classical dynamic control.

Cette thèse propose des contributions très prometteuses au sujet du choix des primitives visuelles en asservissement visuel utilisant les moments 2D extraits de l’image. Nous avons proposé une nouvelle manière de résoudre un problème important en asservissement visuel, à savoir la commande du mouvement de rotation suivant les axes orthogonaux à l’axe optique. Ce travail représente une amélioration significative des travaux précédents en asservissement visuel basé sur l’utilisation des moments 2D extraits de l’image pour commander les degrés de liberté des robots manipulateurs. La commande la plus utilisée est connue sous le nom de commande cinématique. L’approche emploie un descripteur global d’image basé sur des moments 2D "shifted" dont les invariants calculés à partir de ces moments 2D "shifted" utilisent des moments d’ordre faible connus pour être robustes au bruit. De plus, ces invariants choisis ne dépendant pas de la forme de l’objet, sont invariants au mouvement de translation, de rotation et d’échelle. Cette nouvelle façon de faire vient ainsi résoudre les problèmes vus dans les travaux précédents relatifs aux choix des combinaisons des moments invariants basés sur les moments centrés et qui dépendent de la forme de l’objet considéré. Dans les travaux précédents, ces invariants sont calculés avec des moments dont l’ordre est compris entre trois et cinq qui sont vulnérables aux bruits de mesure. D’un point de vue asservissement visuel, le travail est basé sur la détermination explicite de la matrice d’interaction calculée à partir des moments 2D "shifted" dont le choix des paramètres de décalages respectent les propriétés d’invariances aux mouvements de translation, de rotation et d’échelle. En procédant ainsi, les informations visuelles choisies montrent la capacité de représenter les objets de formes symétriques et non symétriques. Des résultats de simulations sont présentés pour illustrer la validité de notre proposition. / This thesis proposes nice and very promising contributions about the choice of visual features in image moments-based visual servoing. We proposed a new way towards solving an important problem in Visual Servoing, namely control of non-optic axis rotational degrees of freedom. This work represents a significant improvement respect to previous works on image-based visual servoing of robot manipulators, with the camera in-hand configuration and under the control scheme known as kinematic control. The approach uses global image features that are based on shifted image moments of a planar target. The improvement consists in a particular selection of a combination of shifted image moments of low order such that they do not depend on the planar target shape ; thus solving the problem seen in related previous works where the selection of central or regular image moments combinations depended on the planar target shapes. From a visual servoing point of view, the work is based on the explicit resolution of the interaction matrix related to any shifted image moment, on the appropriate combination of these moments and on the proper selection of the shifted parameters. By doing so, the new features show improved ability to represent symmetrical objects and several kinds of objects defined from closed contours or from a set of points. Six visual features are selected to design a decoupled control scheme when the object is parallel to the image plane. This nice property is then generalized to the case where the desired object position is not parallel to the image plane. Finally simulated results are presented to illustrate the validity of our proposal.

Uma câmera pan-tilt é capaz de se movimentar em torno de dois eixos de rotação (pan e tilt), permitindo que sua lente possa ser apontada para um ponto qualquer no espaço. Uma aplicação possível dessa câmera é mantê-la apontada para um determinado alvo em movimento, através de posicionamentos angulares pan e tilt adequados. Este trabalho apresenta uma técnica de controle servo visual, em que, inicialmente, as imagens capturadas pela câmera são utilizadas para determinar a posição do alvo. Em seguida, calculam-se as rotações necessárias para manter a projeção do alvo no centro da imagem, em um sistema em tempo real e malha fechada. A técnica de rastreamento visual desenvolvida se baseia em comparação de uma região de referência, utilizando a soma dos quadrados das diferenças (SSD) como critério de correspondência. Sobre essa técnica, é adicionada uma extensão baseada no princípio de estimação incremental e, em seguida, o algoritmo é mais uma vez modificado através do princípio de estimação em multiresolução. Para cada uma das três configurações, são realizados testes para comparar suas performances. O sistema é modelado através do princípio de fluxo óptico e dois controladores são apresentados para realimentar o sistema: um proporcional integral (PI) e um proporcional com estimação de perturbações externas através de um filtro de Kalman (LQG). Ambos são calculados utilizando um critério linear quadrático e os desempenhos deles também são analisados comparativamente. / A pan-tilt camera can move around two rotational axes (pan and tilt), allowing its lens to be pointed to any point in space. A possible application of the camera is to keep it pointed to a certain moving target, through appropriate angular pan-tilt positioning. This work presents a visual servoing technique, which uses first the images captured by the camera to determinate the target position. Then the method calculates the proper rotations to keep the target position in image center, establishing a real-time and closed-loop system. The developed visual tracking technique is based on template region matching, and makes use of the sum of squared differences (SSD) as similarity criterion. An extension based on incremental estimation principle is added to the technique, and then the algorithm is modified again by multiresolution estimation method. Experimental results allow a performance comparison between the three configurations. The system is modeled through optical flow principle and this work presents two controllers to accomplish the system feedback: a proportional integral (PI) and a proportional with external disturbances estimation by a Kalman filter (LQG). Both are determined using a linear quadratic method and their performances are also analyzed comparatively.

Initially an analytical closed-form inverse kinematics solution for a 5 DOF robotic arm was developed and implemented. This analytical solution proved not to meet the accuracy required for the shape sorting puzzle setup used in the COSPAL (COgnitiveSystems using Perception-Action Learning) project [2]. The correctness of the analytic model could be confirmed through a simulated ideal robot and the source of the problem was deemed to be nonlinearities introduced by weak servos unable to compensate for the effect of gravity. Instead of developing a new analytical model that took the effect of gravity into account, which would be erroneous when the characteristics of the robotic arm changed, e.g. when picking up a heavy object, a learning approach was selected. As learning method Locally Weighted Projection Regression (LWPR) [27] is used. It is an incremental supervised learning method and it is considered a state-ofthe-art method for function approximation in high dimensional spaces. LWPR is further combined with visual servoing. This allows for an improvement in accuracy by the use of visual feedback and the problems introduced by the weak servos can be solved. By combining the trained LWPR model with visual servoing, a high level of accuracy is reached, which is sufficient for the shape sorting puzzle setup used in COSPAL.