Orientador: Prof. Dr. João Paulo Gois / Dissertação (mestrado) - Universidade Federal do ABC, Programa de Pós-Graduação em Ciência da Computação, 2014. / The 3D reconstruction of human bodies is used in various areas, for example in medicine, lm, textile industry and the video game industry. Currently there are devices that make use of various technologies for obtaining three-dimensional data of the bodies. More accurate devices have high costs, making it impractical for use in domestic environments or small businesses, in addition to demand complex installations and knowledge of speci c software. Recent advances in the capture of three-dimensional data of objects and interiors have been provided, with applications like KinectFusion, which uses an RGB-D low-cost device. Although accurate for static scenes,these do not reconstruct with precision quasi-rigid objects such as human bodies, which have constant and involuntary movements such as breathing. This dissertation aims to study, develop and evaluate a reconstruction method of human bodies using a RGB-D device low cost, solving the problem of resolution and noise of the data on the device using super-resolution techniques and employing the merging of frames captured by the device, developing registration based on graphs of deformation.

Submitted by Automa??o e Estat?stica (sst@bczm.ufrn.br) on 2016-05-03T22:56:47Z No. of bitstreams: 1 BrunoMarquesFerreiraDaSilva_TESE.pdf: 3729350 bytes, checksum: c9553610a20c907ef1ea2b82c67a5095 (MD5) / Approved for entry into archive by Arlan Eloi Leite Silva (eloihistoriador@yahoo.com.br) on 2016-05-05T19:39:25Z (GMT) No. of bitstreams: 1 BrunoMarquesFerreiraDaSilva_TESE.pdf: 3729350 bytes, checksum: c9553610a20c907ef1ea2b82c67a5095 (MD5) / Made available in DSpace on 2016-05-05T19:39:25Z (GMT). No. of bitstreams: 1 BrunoMarquesFerreiraDaSilva_TESE.pdf: 3729350 bytes, checksum: c9553610a20c907ef1ea2b82c67a5095 (MD5) Previous issue date: 2015-07-31 / Coordena??o de Aperfei?oamento de Pessoal de N?vel Superior (CAPES) / O registro de nuvens de pontos capturadas por sensores de profundidade ? uma importante etapa em aplica??es de reconstru??o 3D. Em diversos casos como localiza??o e mapeamento para rob?tica ou realidade aumentada para entretenimento, o registro deve ser realizado n?o s? com precis?o estrita, como tamb?m na frequ?ncia de dados de aquisi??o do sensor. Com o objetivo de registrar nuvens de pontos de sensores RGB-D (p. ex. Microsoft Kinect), ? proposto nesta tese o uso do algoritmo de fluxo ?ptico piramidal esparso para registro incremental a partir de dados de apar?ncia e profundidade. O erro acumulado inerente ao processo incremental ? posteriormente reduzido, atrav?s do uso de um marcador artificial e minimiza??o de erro por otimiza??o em grafo de poses. Resultados experimentais obtidos ap?s o processamento de diversos conjuntos de dados RGB-D validam o sistema proposto por esta tese para aplica??es de odometria visual, SLAM visual e digitaliza??o de objetos em tempo real. / Registration of point clouds captured by depth sensors is an important task in 3D reconstruction applications based on computer vision. In many applications with strict performance requirements, the registration should be executed not only with precision, but also in the same frequency as data is acquired by the sensor. This thesis proposes theuse of the pyramidal sparse optical flow algorithm to incrementally register point clouds captured by RGB-D sensors (e.g. Microsoft Kinect) in real time. The accumulated errorinherent to the process is posteriorly minimized by utilizing a marker and pose graph optimization. Experimental results gathered by processing several RGB-D datasets validatethe system proposed by this thesis in visual odometry and simultaneous localization and mapping (SLAM) applications.

Nowadays, robot is commonly used to perform automation tasks. With the trend of low volume and customised products, flexible manufacturing is introduced to increase working efficiency and flexibility. Therefore, human robot collaboration plays an important role in automation production and safety should be considered in the design of this kind of robot cell. This work presents the design of safe human robot collaboration by equipping an industrial robot cell with SICK laser scanners, safety monitoring system and trap routine speed control. It also investigates the reliability of RGB-D camera for robot safety. This work aims to find a safety robot system using standard industrial robot for human robot collaboration. The challenge is to ensure the operator's safety at all times. It investigates safety standards and directives, safety requirements of collaboration, and related works for the design of collaborative robot cell, and makes risk assessment before carrying out the valuation. Based on literature review, it gives the concept of layout design and logic for slow down and resume of robot motion. The speed will be first reduced to manual speed and then zero speed depending on the distance between the human and the robot. Valuation and verification are made in the proposed safe solution for human robot collaboration to test the reliability and feasibility. This project realizes the automatic resume that the robot can con-tinue working without manually pressing reset button after the operator leaves the robot cell if there is no access to the prohibited area. In addition, it also adopts the manual reset at the same time to ensure the safety when people access the prohibited area. Several special cases that may happen in the human robot collaboration are described and analysed. Furthermore, the future work is presented to make improvements for the proposed safety robot cell design.

Tato diplomová práce se zabývá tvorbou fúze pozičních dat z existující realtimové im- plementace vizuálního SLAMu a kolové odometrie. Výsledkem spojení dat je potlačení nežádoucích chyb u každé ze zmíněných metod měření, díky čemuž je možné vytvořit přesnější 3D model zkoumaného prostředí. Práce nejprve uvádí teorií potřebnou pro zvládnutí problematiky 3D SLAMu. Dále popisuje vlastnosti použitého open source SLAM projektu a jeho jednotlivé softwarové úpravy. Následně popisuje principy spo- jení pozičních informací získaných vizuálními a odometrickými snímači, dále uvádí popis diferenciálního podvozku, který byl použit pro tvorbu kolové odometrie. Na závěr práce shrnuje výsledky dosažené datovou fúzí a srovnává je s původní přesností vizuálního SLAMu.

Neonatal necrotizing enterocolitis (NEC) is one of the most severe digestive tract emergencies in neonates, involving bowel edema, hemorrhage, and necrosis, and can lead to serious complications including death. Since it is difficult to diagnose early, the morbidity and mortality rates are high due to severe complications in later stages of NEC and thus early detection is key to the treatment of NEC. In this thesis, a novel automatic image acquisition and analysis system combining a color and depth (RGB-D) sensor with an infrared (IR) camera is proposed for NEC diagnosis. A design for sensors configuration and a data acquisition process are introduced. A calibration method between the three cameras is described which aims to ensure frames synchronization and observation consistency among the color, depth, and IR images. Subsequently, complete segmentation procedures based on the original color, depth, and IR information are proposed to automatically separate the human body from the background, remove other interfering items, identify feature points on the human body joints, distinguish the human torso and limbs, and extract the abdominal region of interest. Finally, first-order statistical analysis is performed on thermal data collected over the entire extracted abdominal region to compare differences in thermal data distribution between different patient groups. Experimental validation in a real clinical environment is reported and shows encouraging results.

We investigated the advantage of combining RGB images with depth data to get more robust object classifications and detections using pre-trained deep convolutional neural networks. We relied upon the raw images from publicly available datasets captured using Microsoft Kinect cameras. The raw images varied in size, and therefore required resizing to fit our network. We designed a resizing method called "bleeding edge" to avoid distorting the objects in the images. We present a novel method of interpolating the missing depth pixel values by comparing to similar RGB values. This method proved superior to the other methods tested. We showed that a simple colormap transformation of the depth image can provide close to state-of-art performance. Using our methods, we can present state-of-art performance on the Washington Object dataset and we provide some results on the Washington Scenes (V1) dataset. Specifically, for the detection, we used contours at different thresholds to find the likely object locations in the images. For the classification task we can report state-of-art results using only RGB and RGB-D images, depth data alone gave close to state-of-art results. For the detection task we found the RGB only detector to be superior to the other detectors.

The World Health Organization (WHO) stated in February 2021 at the Seventy- Third World Health Assembly that, globally, at least 2.2 billion people have a near or distance vision impairment. They also denoted the severe impact vision impairment has on the quality of life of the individual suffering from this condition, how it affects the social well-being and their economic independence in society, becoming in some cases an additional burden to also people in their immediate surroundings. In order to minimize the costs and intrusiveness of the applications and maximize the autonomy of the individual life, the natural solution is using systems that rely on computer vision algorithms. The systems improving the quality of life of the visually impaired need to solve different problems such as: localization, path recognition, obstacle detection, environment description, navigation, etc. Each of these topics involves an additional set of problems that have to be solved to address it. For example, for the task of object detection, there is the need of depth prediction to know the distance to the object, path recognition to know if the user is on the road or on a pedestrian path, alarm system to provide notifications of danger for the user, trajectory prediction of the approaching obstacle, and those are only the main key points. Taking a closer look at all of these topics, they have one key component in common: depth estimation/prediction. All of these topics are in need of a correct estimation of the depth in the scenario. In this thesis, our main focus relies on addressing depth estimation in indoor and outdoor environments. Traditional depth estimation methods, like structure from motion and stereo matching, are built on feature correspondences from multiple viewpoints. Despite the effectiveness of these approaches, they need a specific type of data for their proper performance. Since our main goal is to provide systems with minimal costs and intrusiveness that are also easy to handle we decided to infer the depth from single images: monocular depth estimation. Estimating depth of a scene from a single image is a simple task for humans, but it is notoriously more difficult for computational models to be able to achieve high accuracy and low resource requirements. Monocular Depth Estimation is this very task of estimating depth from a single RGB image. Since there is only a need of one image, this approach is used in applications such as autonomous driving, scene understanding or 3D modeling where other type of information is not available. This thesis presents contributions towards solving this task using deep learning as the main tool. The four main contributions of this thesis are: first, we carry out an extensive review of the state-of-the-art in monocular depth estimation; secondly, we introduce a novel large scale high resolution outdoor stereo dataset able to provide enough image information to solve various common computer vision problems; thirdly, we show a set of architectures able to predict monocular depth effectively; and, at last, we propose two real life applications of those architectures, addressing the topic of enhancing the perception for the visually impaired using low-cost wearable sensors.

Made available in DSpace on 2016-12-12T20:22:52Z (GMT). No. of bitstreams: 1 Eduardo B Viecilli.pdf: 12003318 bytes, checksum: 049902e80d65ca85726715d69e30469a (MD5) Previous issue date: 2014-03-17 / Conselho Nacional de Desenvolvimento Científico e Tecnológico / Mobile robots should be able to seek (explore the environment and recognize the objects) autonomously and efficiently. This work developed a mobile robot capable of performing the search for a 3D object in an unknown environment, using only one depth camera (RGB + Depth) as sensor and executing a strategy of active vision. The Microsoft Kinect was adopted as sensor. Also a mobile robot (XKBO) was build using the Robot Operating System (ROS), and with its architecture adapted from the norm STANAG 4586. A new active exploration strategy was developed in which considers the effort of the robot to move to the frontier regions (occult areas), and the presence of traces of the object. The metrics used demonstrated that the use of depth cameras for visual search tasks have a potential for deployment because associates visual and depth information, allowing the robot to better understand the environment and the target object of the search. / Robôs móveis devem ter a capacidade de buscar (explorar o ambiente e reconhecer objetos) de forma autônoma e eficiente. Este trabalho desenvolveu um robô móvel capaz de executar a busca a um objeto (3D) em ambiente desconhecido, utilizando somente uma câmera de profundidade (RGB + distancia) como sensor e executando uma estratégia de visão ativa. A Microsoft Kinect foi a câmera adotada. Também construiu-se um robô móvel (XKBO) que utiliza o Sistema Operacional Robótico (ROS), e com a arquitetura adaptada da norma STANAG 4586. Foi possível usar algoritmos existentes para reconhecer objetos 3D usando o Kinect graças as ferramentas presentes no ROS. E o uso do Kinect facilitou a geração de mapas do ambiente. Desenvolveu-se uma nova estratégia de exploração ativa que considera o esforço de movimentação para as regiões de fronteiras (áreas ocultas), e a existência de indícios da presença do objeto. As métricas utilizadas demonstram que o uso de câmeras de profundidade para tarefas de busca tem potencial para evolução por associar informação visuais com as de profundidade, permitindo que o robô possa entender o ambiente e o objeto alvo da busca. Palavras-chave: Robô Móvel. Exploração. Busca Visual. Câmera RGB-D.

[pt] Esta dissertação investiga o uso das estruturas planares extraídas de imagens RGB-Dem aplicações de Realidade Aumentada. Ter o modelo da cena é fundamental para as aplicações de realidade aumentada. O uso de imagens RGB-D auxilia bastante o processo da construção destes modelos, pois elas fornecem a geometria e os aspectos fotométricos da cena. Devido a grande parte das aplicações de realidade aumentada utilizarem superfícies planares como sua principal componente para projeção de objetos virtuais, é fundamental ter um método robusto e eficaz para obter e representar as estruturas que constituem estas superfícies planares. Neste trabalho, apresentaremos um método para identificar, segmentar e representar estruturas planares a partir de imagens RGB-D. Nossa representação das estruturas planares são polígonos bidimensionais triangulados, simplificados e texturizados, que estão no sistema de coordenadas do plano, onde os pontos destes polígonos definem as regiões deste plano. Demonstramos através de diversos experimentos e da implementação de uma aplicação de realidade aumentada, as técnicas e métodos utilizados para extrair as estruturas planares a partir das imagens RGB-D. / [en] This dissertation investigates the use of planar geometric structures extracted from RGB-D images in Augmented Reality Applications. The model of a scene is essential for augmented reality applications. RGB-D images can greatly help the construction of these models because they provide geometric and photometric information about the scene. Planar structures are prevalent in many 3D scenes and, for this reason, augmented reality applications use planar surfaces as one of the main components for projection of virtual objects. Therefore, it is extremely important to have robust and efficient methods to acquire and represent the structures that compose these planar surfaces. In this work, we will present a method for identifying, targeting and representing planar structures from RGB-D images. Our planar structures representation is triangulated two-dimensional polygons, simplified and textured, forming a triangle mesh intrinsic to the plane that defines regions in this space corresponding to surface of objects in the 3D scene. We have demonstrated through various experiments and implementation of an augmented reality application, the techniques and methods used to extract the planar structures from the RGB-D images.

Ce manuscrit propose une approche méthodologique pour la constitution d’une chaîne complète de vidéosurveillance pour des environnements naturellement encombrés. Nous identifions et levons un certain nombre de verrous méthodologiques et technologiques inhérents : 1) à l’acquisition de séquences vidéo en milieu naturel, 2) au traitement d’images, 3) au suivi multi-cibles, 4) à la découverte et la modélisation de motifs comportementaux récurrents, et 5) à la fusion de données. Le contexte applicatif de nos travaux est la surveillance apicole, et en particulier, l’étude des trajectoires des abeilles en vol devant la ruche. De ce fait, cette thèse se présente également comme une étude de faisabilité et de prototypage dans le cadre des deux projets interdisciplinaires EPERAS et RISQAPI (projets menées en collaboration avec l’INRA Magneraud et le Muséum National d’Histoire Naturelle). Il s’agit pour nous informaticiens et pour les biologistes qui nous ont accompagnés, d’un domaine d’investigation totalement nouveau, pour lequel les connaissances métiers, généralement essentielles à ce genre d’applications, restent encore à définir. Contrairement aux approches existantes de suivi d’insectes, nous proposons de nous attaquer au problème dans l’espace à trois dimensions grâce à l’utilisation d’une caméra stéréovision haute fréquence. Dans ce contexte, nous détaillons notre nouvelle méthode de détection de cibles appelée segmentation HIDS. Concernant le calcul des trajectoires, nous explorons plusieurs approches de suivi de cibles, s’appuyant sur plus ou moins d’a priori, susceptibles de supporter les conditions extrêmes de l’application (e.g. cibles nombreuses, de petite taille, présentant un mouvement chaotique). Une fois les trajectoires collectées, nous les organisons selon une structure de données hiérarchique et mettons en œuvre une approche Bayésienne non-paramétrique pour la découverte de comportements émergents au sein de la colonie d’insectes. L’analyse exploratoire des trajectoires issues de la scène encombrée s’effectue par classification non supervisée, simultanément sur des niveaux sémantiques différents, et où le nombre de clusters pour chaque niveau n’est pas défini a priori mais est estimé à partir des données. Cette approche est dans un premier temps validée à l’aide d’une pseudo-vérité terrain générée par un Système Multi-Agents, puis dans un deuxième temps appliquée sur des données réelles. / This manuscript provides the basis for a complete chain of videosurveillence for naturally cluttered environments. In the latter, we identify and solve the wide spectrum of methodological and technological barriers inherent to : 1) the acquisition of video sequences in natural conditions, 2) the image processing problems, 3) the multi-target tracking ambiguities, 4) the discovery and the modeling of recurring behavioral patterns, and 5) the data fusion. The application context of our work is the monitoring of honeybees, and in particular the study of the trajectories bees in flight in front of their hive. In fact, this thesis is part a feasibility and prototyping study carried by the two interdisciplinary projects EPERAS and RISQAPI (projects undertaken in collaboration with INRA institute and the French National Museum of Natural History). It is for us, computer scientists, and for biologists who accompanied us, a completely new area of investigation for which the scientific knowledge, usually essential for such applications, are still in their infancy. Unlike existing approaches for monitoring insects, we propose to tackle the problem in the three-dimensional space through the use of a high frequency stereo camera. In this context, we detail our new target detection method which we called HIDS segmentation. Concerning the computation of trajectories, we explored several tracking approaches, relying on more or less a priori, which are able to deal with the extreme conditions of the application (e.g. many targets, small in size, following chaotic movements). Once the trajectories are collected, we organize them according to a given hierarchical data structure and apply a Bayesian nonparametric approach for discovering emergent behaviors within the colony of insects. The exploratory analysis of the trajectories generated by the crowded scene is performed following an unsupervised classification method simultaneously over different levels of semantic, and where the number of clusters for each level is not defined a priori, but rather estimated from the data only. This approach is has been validated thanks to a ground truth generated by a Multi-Agent System. Then we tested it in the context of real data.