Dans ce travail, nous proposons une représentation efficace de l’environnement adaptée à la problématique de la navigation autonome. Cette représentation topométrique est constituée d’un graphe de sphères de vision augmentées d’informations de profondeur. Localement la sphère de vision augmentée constitue une représentation égocentrée complète de l’environnement proche. Le graphe de sphères permet de couvrir un environnement de grande taille et d’en assurer la représentation. Les "poses" à 6 degrés de liberté calculées entre sphères sont facilement exploitables par des tâches de navigation en temps réel. Dans cette thèse, les problématiques suivantes ont été considérées : Comment intégrer des informations géométriques et photométriques dans une approche d’odométrie visuelle robuste ; comment déterminer le nombre et le placement des sphères augmentées pour représenter un environnement de façon complète ; comment modéliser les incertitudes pour fusionner les observations dans le but d’augmenter la précision de la représentation ; comment utiliser des cartes de saillances pour augmenter la précision et la stabilité du processus d’odométrie visuelle. / Our aim is concentrated around building ego-centric topometric maps represented as a graph of keyframe nodes which can be efficiently used by autonomous agents. The keyframe nodes which combines a spherical image and a depth map (augmented visual sphere) synthesises information collected in a local area of space by an embedded acquisition system. The representation of the global environment consists of a collection of augmented visual spheres that provide the necessary coverage of an operational area. A "pose" graph that links these spheres together in six degrees of freedom, also defines the domain potentially exploitable for navigation tasks in real time. As part of this research, an approach to map-based representation has been proposed by considering the following issues : how to robustly apply visual odometry by making the most of both photometric and ; geometric information available from our augmented spherical database ; how to determine the quantity and optimal placement of these augmented spheres to cover an environment completely ; how tomodel sensor uncertainties and update the dense infomation of the augmented spheres ; how to compactly represent the information contained in the augmented sphere to ensure robustness, accuracy and stability along an explored trajectory by making use of saliency maps.

Simultaneous Localization and Mapping (SLAM) is the crucial ability for many autonomous systems to operate in unknown environments. In recent years SLAM development has focused on achieving robustness regarding the challenges the field still faces e.g. dynamic environments. During this thesis work different existing approaches to tackle dynamics with Visual SLAM systems were analyzed by surveying the recent literature within the field. The goal was to define the advantages and drawbacks of the approaches to provide further insight into the field of dynamic SLAM. Furthermore, two methods of different approaches were chosen for experiments and their implementation was documented. Key conclusions from the literature survey and experiments are the following. The exclusion of dynamic objects with regard to camera pose estimation presents promising results. Tracking of dynamic objects provides valuable information when combining SLAM with other tasks e.g. path planning. Moreover, dynamic reconstruction with SLAM offers better scene understanding and analysis of objects’ behavior within an environment. Many solutions rely on pre-processing and heavy hardware requirements due to the nature of the object detection methods. Methods of motion confirmation of objects lack consideration of camera movement, resulting in static objects being excluded from feature extraction. Considerations for future work within the field include accounting for camera movement for motion confirmation and producing available benchmarks that offer evaluation of the SLAM result as well as the dynamic object detection i.e. ground truth for both camera and objects within the scene. / Simultaneous Localization and Mapping (SLAM) är för många autonoma system avgörande för deras förmåga att kunna verka i tidigare outforskade miljöer. Under de senaste åren har SLAM-utvecklingen fokuserat på att uppnå robusthet när det gäller de utmaningar som fältet fortfarande står inför, t.ex. dynamiska miljöer. I detta examensarbete analyserades befintliga metoder för att hantera dynamik med visuella SLAM-system genom att kartlägga den senaste litteraturen inom området. Målet var att definiera för- och nackdelar hos de olika tillvägagångssätten för att bidra med insikter till området dynamisk SLAM. Dessutom valdes två metoder från olika tillvägagångssätt ut för experiment och deras implementering dokumenterades. De viktigaste slutsatserna från litteraturstudien och experimenten är följande. Uteslutningen av dynamiska objekt vid uppskattning av kamerans position ger lovande resultat. Spårning av dynamiska objekt ger värdefull information när SLAM kombineras med andra uppgifter, t.ex. path planning. Dessutom ger dynamisk rekonstruktion med SLAM bättre förståelse om omgivningen och analys av objekts beteende i den kringliggande miljön. Många lösningar är beroende av förbehandling samt ställer höga hårdvarumässiga krav till följd av objektdetekteringsmetodernas natur. Metoder för rörelsebekräftelse av objekt tar inte hänsyn till kamerarörelser, vilket leder till att statiska objekt utesluts från funktionsextraktion. Uppmaningar för framtida studier inom området inkluderar att ta hänsyn till kamerarörelser under rörelsebekräftelse samt att ta ändamålsenliga riktmärken för att möjliggöra tydligare utvärdering av SLAM-resultat såväl som för dynamisk objektdetektion, dvs. referensvärden för både kamerans position såväl som för objekt i scenen.

Tesis por compendio / [ES] A mediados del siglo XX inició en el continente europeo, especialmente en Francia y Bélgica, una nueva disciplina denominada Ergonomía Centrada en la Actividad. Esta disciplina está enfocada en el análisis del trabajo con el fin de optimizar las condiciones laborales. El propósito de la intervención ergonómica es mejorar componentes que interactúan en el sistema o en la actividad del trabajo (las personas, la organización, la tecnología y el ambiente), interrelacionando aspectos de salud, seguridad, productividad y calidad. En investigaciones anteriores a la presente el doctorando identificó la necesidad de mejora de las herramientas empleadas por los profesionales para la evaluación de factores de riesgo ergonómico, motivando así el desarrollo de la presente investigación. Inicialmente se analizaron métodos generales empleados por los diseñadores para establecer aquellos susceptibles de mejora con la introducción de nuevas tecnologías. En una segunda fase se identificaron y analizaron dispositivos tecnológicos orientados al rastreo de la actividad humana aplicables en el ámbito de las metodologías de la ergonomía ocupacional para los sectores productivos. Como resultado se concluyó que los sensores RGB-D y el Eye-Tracking (rastreo ocular) son dispositivos aplicables para ayudar a mejorar las condiciones en los puestos de trabajo, el primero para distribuir las áreas de trabajo y el segundo para la mejora de las interfaces del usuario. En esta TD se desarrollaron técnicas y métodos para el empleo de estos dispositivos logrando el diseño ergonómico de puestos de trabajo con aplicación práctica (artículos 2 y 3). Durante la realización de esta TD, bajo las directrices del Programa de Doctorado en Tecnologías para la Salud y el Bienestar de la Universidad Politécnica de Valencia, se publicaron tres artículos en revistas que durante el año de su publicación estuvieron indexadas en el primer cuartil de su categoría en el Journal Citation Report, las cuáles sustentan los resultados de la investigación. En ellos se evidenció cómo los avances tecnológicos implementados en Ergonomía producen cambios importantes en el diseño de los puestos de trabajo, y minimizan los tiempos y los movimientos que se requieren en las diferentes actividades laborales, garantizando así una ubicación óptima del recurso humano en los sistemas de producción y generando a su vez estrategias que disminuyen los Trastornos Músculo Esqueléticos (TMEs). / [CA] A mitjan segle XX es va desenvolupar en el continent europeu, especialment a França i Bèlgica, una nova disciplina denominada Ergonomia Centrada en l'Activitat. Aquesta disciplina està enfocada en l'anàlisi del treball amb la finalitat d'optimitzar les condicions laborals. El propòsit de la intervenció ergonòmica és millorar els components que interactuen en el sistema o en l'activitat del treball (les persones, l'organització, la tecnologia i l'ambient), interrelacionant els aspectes de salut, seguretat, productivitat i qualitat. En investigacions anteriors a la present el doctorand va identificar la necessitat de millora de les eines emprades pels professionals per a l'avaluació de factors de risc ergonòmic, motivant així el desenvolupament de la present investigació. Inicialment es van analitzar els mètodes habituals empleats pels ergónoms per a establir aquells susceptibles de millora amb la introducció de noves tecnologies. En una segona fase es va identificar i analitzar els dispositius tecnològics orientats al monitoratge de l'activitat humana aplicables en l'àmbit de les metodologies de l'ergonomia ocupacional per als sectors productius. Com a resultat es va concloure que els sensors RGB-D i el Eye-Tracking són dispositius aplicables per a ajudar a millorar les condicions en els llocs de treball, el primer per a distribuir les àrees de treball i el segon per a la millora de les interfícies de l'usuari. En aquesta TD es van desenvolupar tècniques i procediments per a l'ús d'aquests dispositius aconseguint el disseny ergonòmic de llocs de treball amb aplicació pràctica (articles 2 i 3). Durant la realització d'aquesta TD, sota les directrius del Programa de Doctorat en Tecnologies per a la Salut i el Benestar de la Universitat Politècnica de València, es van publicar tres articles en revistes, que durant l'any de la seua publicació, van estar indexades en el primer quartil de la seua categoria en el Journal Citation Report, les quals sustenten els resultats de la investigació. En ells es va evidenciar com els avanços tecnològics implementats en Ergonomia produeixen canvis importants en el disseny dels llocs de treball, minimitzen els temps i els moviments que es requereixen en les diferents activitats laborals, garantint així una ubicació òptima del recurs humà en els sistemes de producció i generant al seu torn estratègies que disminueixen els Trastorns Músculesquelètics. / [EN] Around the middle of XX century in Europe, especially in France and Belgium, a new discipline named Activity Focused Ergonomics was developed. This concept is based on the task analysis aiming to optimize working conditions. The purpose of the ergonomic intervention is to improve the components interacting within the system or the work activities (people, organization, technology and environment), correlating all health, safety, productivity and quality facts. In previous research, the ergomists identified the improvement need of those tools used by professionals to assess ergonomic risk factors, enhancing therefore the development of this present research. In the beginning, regular methods used by ergonomics specilists were analized in order to define those that could be improved with the introduction of new technologies. In a second phase, technological devices aimed to monitorize human activity, as well as those applicable for the occupational ergonomics methodologies in productive sectors were identified and analyzed. The conclusion that came as a result was that RGB-D sensors and Eye-Tracking are actual workstations conditions' improving devices, the first one is used to better organize working areas and the second one to optimize the user's interfaces. This Doctoral Thesis develops techniques and procedures to correctly use these devices obtaining workstations ergonomic designs with practical applications (articles 2 and 3). Along the development of this Doctoral Thesis, under the Polytechnic University of Valencia Doctoral School for Health and Wellbeing Technologies program directions, three articles were published in magazines which, throughout their publication year were indexed within its cathegory's first quartile at the Journal Citation Report, supporting the research results. The articles evidenced how the technological progress implemented in Ergonomics produce important changes in the workstations design and minimize times and movements required in different working activities, ensuring human resources' optimal location within the production systems and also developing Musculoskeletal Disorders reduction strategies. / Garzón Leal, DC. (2020). Nuevas tecnologías aplicadas a la ergonomía ocupacional. Empleo de sensores RGBD y EyeTracking en la mejora ergonómica de puestos de trabajo [Tesis doctoral]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/156052 / Compendio

Sistemas de Interação Humano-Computador baseados em toque são uma tecnologia disseminada em tablets, smartphones e notebooks. Trata-se de um grande avanço que aumenta a facilidade de comunicação e, ao mesmo tempo, diminui a necessidade de interfaces como mouse e teclado. Entretanto, a superfície de interação utilizada por esses sistemas normalmente é equipada com sensores para a captação dos movimentos realizados pelo usuário, o que impossibilita transformar uma superfície planar qualquer (uma mesa, por exemplo) em uma superfície de interação. Por outro lado, a popularização de sensores de profundidade a partir do lançamento do Microsoft Kinect propiciou o desenvolvimento de sistemas que adotam objetos do dia a dia como superfícies de interação. Nesta dissertação é proposta uma interface natural para interação com superfícies planares utilizando uma câmera RGB-D em posição descendente. Inicialmente, o plano de interação é localizado na nuvem de pontos 3D através de uma variação do algoritmo RANSAC com coerência temporal. Objetos acima do plano são segmentados a partir da transformada watershed baseada em uma função de energia que combina cor, profundidade e informação de confiança. A cor de pele é utilizada para isolar as mãos, e os dedos que interagem com o plano são identificados por um novo processo de esqueletonização 2D. Finalmente, as pontas dos dedos são rastreadas com o uso do algoritmo Húngaro, e o filtro de Kalman é usado para produzir trajetórias mais suaves. Para demonstrar a utilidade da técnica, foi desenvolvido um protótipo que permite ao usuário desenhar em uma superfície de forma natural e intuitiva. / Touch-based Human-Computer Interfaces (HCIs) are a widespread technology present in tablets, smartphones, and notebooks. This is a breakthrough which increases the ease of communication and at the same time reduces the need for interfaces such as mouse and keyboard. However, the interaction surface used by these systems is usually equipped with sensors to capture the movements made by the user, making it impossible to substitute this surface by any other such as a table, for example. On the other hand, the progress of commercial 3D depth sensing technologies in the past five years, having as a keystone Microsoft’s Kinect sensor, has increased the interest in 3D hand gesture recognition using depth data. In this dissertation, we present a natural Human-Computer Interface (HCI) for interaction with planar surfaces using a topdown RGB-D camera. Initially, the interaction plane is located in the 3D point cloud by using a variation of RANSAC with temporal coherence. Off-plane objects are segmented using the watershed transform based on an energy function that combines color, depth and confidence information. Skin color information is used to isolate the hand(s), and a novel 2D skeletonization process identifies the interaction fingers. Finally, the fingertips are tracked using the Hungarian algorithm, and a Kalman filter is applied to produce smoother trajectories. To demonstrate the usefulness of the technique, we also developed a prototype in which the user can draw on the surface using lines and sprays in a natural way.

In this thesis a methodology for representing 3D subjects and their deformations in adverse situations is studied. The study is focused in providing methods based on registration techniques to improve the data in situations where the sensor is working in the limit of its sensitivity. In order to do this, it is proposed two methods to overcome the problems which can difficult the process in these conditions. First a rigid registration based on model registration is presented, where the model of 3D planar markers is used. This model is estimated using a proposed method which improves its quality by taking into account prior knowledge of the marker. To study the deformations, it is proposed a framework to combine multiple spaces in a non-rigid registration technique. This proposal improves the quality of the alignment with a more robust matching process that makes use of all available input data. Moreover, this framework allows the registration of multiple spaces simultaneously providing a more general technique. Concretely, it is instantiated using colour and location in the matching process for 3D location registration.

Dans ce travail, méthodes d'estimation basées sur des images, également connu sous le nom de méthodes directes, sont étudiées qui permettent d'éviter l'extraction de caractéristiques et l'appariement complètement. L'objectif est de produire pose 3D précis et des estimations de la structure. Les fonctions de coût présenté minimiser l'erreur du capteur, car les mesures ne sont pas transformés ou modifiés. Dans la caméra photométrique estimation de la pose, rotation 3D et les paramètres de traduction sont estimées en minimisant une séquence de fonctions de coûts à base d'image, qui sont des non-linéaires en raison de la perspective projection et la distorsion de l'objectif. Dans l'image la structure basée sur le raffinement, d'autre part, de la structure 3D est affinée en utilisant un certain nombre de vues supplémentaires et un coût basé sur l'image métrique. Les principaux domaines d'application dans ce travail sont des reconstitutions d'intérieur, la robotique et la réalité augmentée. L'objectif global du projet est d'améliorer l'image des méthodes d'estimation fondées, et pour produire des méthodes de calcul efficaces qui peuvent être accueillis dans des applications réelles. Les principales questions pour ce travail sont : Qu'est-ce qu'une formulation efficace pour une image 3D basé estimation de la pose et de la structure tâche de raffinement ? Comment organiser calcul afin de permettre une mise en œuvre efficace en temps réel ? Quelles sont les considérations pratiques utilisant l'image des méthodes d'estimation basées sur des applications telles que la réalité augmentée et la reconstruction 3D ?

Video self modeling (VSM) is a behavioral intervention technique in which a learner models a target behavior by watching a video of oneself. This is the idea behind the psychological theory of self-efficacy - you can learn or model to perform certain tasks because you see yourself doing it, which provides the most ideal form of behavior modeling. The effectiveness of VSM has been demonstrated for many different types of disabilities and behavioral problems ranging from stuttering, inappropriate social behaviors, autism, selective mutism to sports training. However, there is an inherent difficulty associated with the production of VSM material. Prolonged and persistent video recording is required to capture the rare, if not existed at all, snippets that can be used to string together in forming novel video sequences of the target skill. To solve this problem, in this dissertation, we use computational multimedia techniques to facilitate the creation of synthetic visual content for self-modeling that can be used by a learner and his/her therapist with a minimum amount of training data. There are three major technical contributions in my research. First, I developed an Adaptive Video Re-sampling algorithm to synthesize realistic lip-synchronized video with minimal motion jitter. Second, to denoise and complete the depth map captured by structure-light sensing systems, I introduced a layer based probabilistic model to account for various types of uncertainties in the depth measurement. Third, I developed a simple and robust bundle-adjustment based framework for calibrating a network of multiple wide baseline RGB and depth cameras.

Sistemas de Interação Humano-Computador baseados em toque são uma tecnologia disseminada em tablets, smartphones e notebooks. Trata-se de um grande avanço que aumenta a facilidade de comunicação e, ao mesmo tempo, diminui a necessidade de interfaces como mouse e teclado. Entretanto, a superfície de interação utilizada por esses sistemas normalmente é equipada com sensores para a captação dos movimentos realizados pelo usuário, o que impossibilita transformar uma superfície planar qualquer (uma mesa, por exemplo) em uma superfície de interação. Por outro lado, a popularização de sensores de profundidade a partir do lançamento do Microsoft Kinect propiciou o desenvolvimento de sistemas que adotam objetos do dia a dia como superfícies de interação. Nesta dissertação é proposta uma interface natural para interação com superfícies planares utilizando uma câmera RGB-D em posição descendente. Inicialmente, o plano de interação é localizado na nuvem de pontos 3D através de uma variação do algoritmo RANSAC com coerência temporal. Objetos acima do plano são segmentados a partir da transformada watershed baseada em uma função de energia que combina cor, profundidade e informação de confiança. A cor de pele é utilizada para isolar as mãos, e os dedos que interagem com o plano são identificados por um novo processo de esqueletonização 2D. Finalmente, as pontas dos dedos são rastreadas com o uso do algoritmo Húngaro, e o filtro de Kalman é usado para produzir trajetórias mais suaves. Para demonstrar a utilidade da técnica, foi desenvolvido um protótipo que permite ao usuário desenhar em uma superfície de forma natural e intuitiva. / Touch-based Human-Computer Interfaces (HCIs) are a widespread technology present in tablets, smartphones, and notebooks. This is a breakthrough which increases the ease of communication and at the same time reduces the need for interfaces such as mouse and keyboard. However, the interaction surface used by these systems is usually equipped with sensors to capture the movements made by the user, making it impossible to substitute this surface by any other such as a table, for example. On the other hand, the progress of commercial 3D depth sensing technologies in the past five years, having as a keystone Microsoft’s Kinect sensor, has increased the interest in 3D hand gesture recognition using depth data. In this dissertation, we present a natural Human-Computer Interface (HCI) for interaction with planar surfaces using a topdown RGB-D camera. Initially, the interaction plane is located in the 3D point cloud by using a variation of RANSAC with temporal coherence. Off-plane objects are segmented using the watershed transform based on an energy function that combines color, depth and confidence information. Skin color information is used to isolate the hand(s), and a novel 2D skeletonization process identifies the interaction fingers. Finally, the fingertips are tracked using the Hungarian algorithm, and a Kalman filter is applied to produce smoother trajectories. To demonstrate the usefulness of the technique, we also developed a prototype in which the user can draw on the surface using lines and sprays in a natural way.

Sistemas de Interação Humano-Computador baseados em toque são uma tecnologia disseminada em tablets, smartphones e notebooks. Trata-se de um grande avanço que aumenta a facilidade de comunicação e, ao mesmo tempo, diminui a necessidade de interfaces como mouse e teclado. Entretanto, a superfície de interação utilizada por esses sistemas normalmente é equipada com sensores para a captação dos movimentos realizados pelo usuário, o que impossibilita transformar uma superfície planar qualquer (uma mesa, por exemplo) em uma superfície de interação. Por outro lado, a popularização de sensores de profundidade a partir do lançamento do Microsoft Kinect propiciou o desenvolvimento de sistemas que adotam objetos do dia a dia como superfícies de interação. Nesta dissertação é proposta uma interface natural para interação com superfícies planares utilizando uma câmera RGB-D em posição descendente. Inicialmente, o plano de interação é localizado na nuvem de pontos 3D através de uma variação do algoritmo RANSAC com coerência temporal. Objetos acima do plano são segmentados a partir da transformada watershed baseada em uma função de energia que combina cor, profundidade e informação de confiança. A cor de pele é utilizada para isolar as mãos, e os dedos que interagem com o plano são identificados por um novo processo de esqueletonização 2D. Finalmente, as pontas dos dedos são rastreadas com o uso do algoritmo Húngaro, e o filtro de Kalman é usado para produzir trajetórias mais suaves. Para demonstrar a utilidade da técnica, foi desenvolvido um protótipo que permite ao usuário desenhar em uma superfície de forma natural e intuitiva. / Touch-based Human-Computer Interfaces (HCIs) are a widespread technology present in tablets, smartphones, and notebooks. This is a breakthrough which increases the ease of communication and at the same time reduces the need for interfaces such as mouse and keyboard. However, the interaction surface used by these systems is usually equipped with sensors to capture the movements made by the user, making it impossible to substitute this surface by any other such as a table, for example. On the other hand, the progress of commercial 3D depth sensing technologies in the past five years, having as a keystone Microsoft’s Kinect sensor, has increased the interest in 3D hand gesture recognition using depth data. In this dissertation, we present a natural Human-Computer Interface (HCI) for interaction with planar surfaces using a topdown RGB-D camera. Initially, the interaction plane is located in the 3D point cloud by using a variation of RANSAC with temporal coherence. Off-plane objects are segmented using the watershed transform based on an energy function that combines color, depth and confidence information. Skin color information is used to isolate the hand(s), and a novel 2D skeletonization process identifies the interaction fingers. Finally, the fingertips are tracked using the Hungarian algorithm, and a Kalman filter is applied to produce smoother trajectories. To demonstrate the usefulness of the technique, we also developed a prototype in which the user can draw on the surface using lines and sprays in a natural way.

In this thesis we study the problem of multi-session dense rgb-d slam for 3D reconstruc- tion. Multi-session reconstruction can allow users to capture parts of an object that could not easily be captured in one session, due for instance to poor accessibility or user mistakes. We first present a thorough overview of single-session dense rgb-d slam and describe the multi-session problem as a loosening of the incremental camera movement and static scene assumptions commonly held in the single-session case. We then implement and evaluate sev- eral variations on a system for doing two-session reconstruction as an extension to a single- session dense rgb-d slam system. The extension from one to several sessions is divided into registering separate sessions into a single reference frame, re-optimizing the camera trajectories, and fusing together the data to generate a final 3D model. Registration is done by matching reconstructed models from the separate sessions using one of two adaptations on a 3D object detection pipeline. The registration pipelines are evaluated with many different sub-steps on a challenging dataset and it is found that robust registration can be achieved using the proposed methods on scenes without degenerate shape symmetry. In particular we find that using plane matches between two sessions as constraints for as much as possible of the registration pipeline improves results. Several different strategies for re-optimizing camera trajectories using data from both ses- sions are implemented and evaluated. The re-optimization strategies are based on re- tracking the camera poses from all sessions together, and then optionally optimizing over the full problem as represented on a pose-graph. The camera tracking is done by incrementally building and tracking against a tsdf volume, from which a final 3D mesh model is extracted. The whole system is qualitatively evaluated against a realistic dataset for multi-session re- construction. It is concluded that the overall approach is successful in reconstructing objects from several sessions, but that other fine grained registration methods would be required in order to achieve multi-session reconstructions that are indistinguishable from singe-session results in terms of reconstruction quality.