Modern medical imaging devices are able to generate highly detailed three-dimensional (3D) images of the skeleton. Computerized image processing and analysis methods, combined with real-time volume visualization techniques, can greatly facilitate the interpretation of such images and are increasingly used in surgical planning to aid reconstruction of the skeleton after trauma or disease. Two key challenges are to accurately separate (segment) bone structures or cavities of interest from the rest of the image and to interact with the 3D data in an efficient way. This thesis presents efficient and precise interactive methods for segmenting, visualizing, and analysing 3D computed tomography (CT) images of the skeleton. The methods are validated on real CT datasets and are primarily intended to support planning and evaluation of cranio-maxillofacial (CMF) and orthopedic surgery. Two interactive methods for segmenting the orbit (eye-socket) are introduced. The first method implements a deformable model that is guided and fitted to the orbit via haptic 3D interaction, whereas the second method implements a user-steered volumetric brush that uses distance and gradient information to find exact object boundaries. The thesis also presents a semi-automatic method for measuring 3D angulation changes in wrist fractures. The fractured bone is extracted with interactive mesh segmentation, and the angulation is determined with a technique based on surface registration and RANSAC. Lastly, the thesis presents an interactive and intuitive tool for segmenting individual bones and bone fragments. This type of segmentation is essential for virtual surgery planning, but takes several hours to perform with conventional manual methods. The presented tool combines GPU-accelerated random walks segmentation with direct volume rendering and interactive 3D texture painting to enable quick marking and separation of bone structures. It enables the user to produce an accurate segmentation within a few minutes, thereby removing a major bottleneck in the planning procedure.

Modern medical imaging techniques provide an increasing amount of high-dimensional and high-resolution image data that need to be visualized, analyzed, and interpreted for diagnostic and treatment planning purposes. As a consequence, efficient ways of exploring these images are needed. In order to work with specific patient cases, it is necessary to be able to work directly with the medical image volumes and to generate the relevant 3D structures directly as they are needed for visualization and analysis. This requires efficient tools for segmentation, i.e., separation of objects from each other and from the background. Segmentation is hard to automate due to, e.g., high shape variability of organs and limited contrast between tissues. Manual segmentation, on the other hand, is tedious and error-prone. An approach combining the merits from automatic and manual methods is semi-automatic segmentation, where the user interactively provides input to the methods. For complex medical image volumes, the interactive part can be highly 3D oriented and is therefore dependent on the user interface. This thesis presents methods for interactive segmentation and visualization where true 3D interaction with haptic feedback and stereo graphics is used. Well-known segmentation methods such as fast marching, fuzzy connectedness, live-wire, and deformable models, have been tailored and extended for implementation in a 3D environment where volume visualization and haptics are used to guide the user. The visualization is accelerated with graphics hardware and therefore allows for volume rendering in stereo at interactive rates. The haptic feedback is rendered with constraint-based direct volume haptics in order to convey information about the data that is hard to visualize and thereby facilitate the interaction. The methods have been applied to real medical images, e.g., 3D liver CT data and 4D breast MR data with good results. To provide a tool for future work in this area, a software toolkit containing the implementations of the developed methods has been made publicly available.

The objective of the proposed research is to develop methods that couple an expert user's guidance with automatic image segmentation and registration algorithms. Often, complex processes such as fire, anatomical changes/variations in human bodies, or unpredictable human behavior produce the target images; in these cases, creating a model that precisely describes the process is not feasible. A common solution is to make simplifying assumptions when performing detection, segmentation, or registration tasks automatically. However, when these assumptions are not satisfied, the results are unsatisfactory. Hence, removing these, often times stringent, assumptions at the cost of minimal user input is considered an acceptable trade-off. Three milestones towards reaching this goal have been achieved. First, an interactive image segmentation approach was created in which the user is coupled in a closed-loop control system with a level set segmentation algorithm. The user's expert knowledge is combined with the speed of automatic segmentation. Second, a stochastic point set registration algorithm is presented. The point sets can be derived from simple user input (e.g. a thresholding operation), and time consuming correspondence labeling is not required. Furthermore, common smoothness assumptions on the non-rigid deformation field are removed. Third, a stochastic image registration algorithm is designed to capture large misalignments. For future research, several improvements of the registration are proposed, and an iterative, landmark based segmentation approach, which couples the segmentation and registration, is envisioned.

La segmentation est l'un des principaux thèmes du domaine de l'analyse d'images. Segmenter une image consiste à trouver une partition constituée de régions, c'est-à-dire d'ensembles de pixels connexes homogènes selon un critère choisi. L'objectif de la segmentation consiste à obtenir des régions correspondant aux objets ou aux parties des objets qui sont présents dans l'image et dont la nature dépend de l'application visée. Même s'il peut être très fastidieux, un tel découpage de l'image peut être facilement obtenu par un être humain. Il n'en est pas de même quand il s'agit de créer un programme informatique dont l'objectif est de segmenter les images de manière entièrement automatique. La segmentation interactive est une approche semi-automatique où l'utilisateur guide la segmentation d'une image en donnant des indications. Les méthodes qui s'inscrivent dans cette approche se divisent en deux catégories en fonction de ce qui est recherché : les contours ou les régions. Les méthodes qui recherchent des contours permettent d'extraire un unique objet correspondant à une région sans trou. L'utilisateur vient guider la méthode en lui indiquant quelques points sur le contour de l'objet. L'algorithme se charge de relier chacun des points par une courbe qui respecte les caractéristiques de l'image (les pixels de part et d'autre de la courbe sont aussi dissemblables que possible), les indications données par l'utilisateur (la courbe passe par chacun des points désignés) et quelques propriétés intrinsèques (les courbes régulières sont favorisées). Les méthodes qui recherchent les régions groupent les pixels de l'image en des ensembles, de manière à maximiser la similarité en leur sein et la dissemblance entre les différents ensembles. Chaque ensemble correspond à une ou plusieurs composantes connexes et peut contenir des trous. L'utilisateur guide la méthode en traçant des traits de couleur qui désignent quelques pixels appartenant à chacun des ensembles. Si la majorité des méthodes ont été conçues pour extraire un objet principal du fond, les travaux menés durant la dernière décennie ont permis de proposer des méthodes dites multiclasses, capables de produire une partition de l'image en un nombre arbitraire d'ensembles. La contribution principale de ce travail de recherche est la conception d'une nouvelle méthode de segmentation interactive multiclasse par recherche des régions. Elle repose sur la modélisation du problème comme la minimisation d'une fonction de coût pouvant être représentée par un graphe de facteurs. Elle intègre une méthode de classification par apprentissage supervisé assurant l'adéquation entre la segmentation produite et les indications données par l'utilisateur, l'utilisation d'un nouveau terme de régularisation et la réalisation d'un prétraitement consistant à regrouper les pixels en petites régions cohérentes : les superpixels. L'utilisation d'une méthode de sur-segmentation produisant des superpixels est une étape clé de la méthode que nous proposons : elle réduit considérablement la complexité algorithmique et permet de traiter des images contenant plusieurs millions de pixels, tout en garantissant un temps interactif. La seconde contribution de ce travail est une évaluation des algorithmes permettant de grouper les pixels en superpixels, à partir d'un nouvel ensemble de données de référence que nous mettons à disposition et dont la particularité est de contenir des images de tailles différentes : de quelques milliers à plusieurs millions de pixels. Cette étude nous a également permis de concevoir et d'évaluer une nouvelle méthode de production de superpixels. / Image segmentation is one of the main research topics in image analysis. It is the task of researching a partition into regions, i.e., into sets of connected pixels, meeting a given uniformity criterion. The goal of image segmentation is to find regions corresponding to the objects or the object parts appearing in the image. The choice of what objects are relevant depends on the application context. Manually locating these objects is a tedious but quite simple task. Designing an automatic algorithm able to achieve the same result is, on the contrary, a difficult problem. Interactive segmentation methods are semi-automatic approaches where a user guide the search of a specific segmentation of an image by giving some indications. There are two kinds of methods : boundary-based and region-based interactive segmentation methods. Boundary-based methods extract a single object corresponding to a unique region without any holes. The user guides the method by selecting some boundary points of the object. The algorithm search for a curve linking all the points given by the user, following the boundary of the object and having some intrinsic properties (regular curves are encouraged). Region-based methods group the pixels of an image into sets, by maximizing the similarity of pixels inside each set and the dissimilarity between pixels belonging to different sets. Each set can be composed of one or several connected components and can contain holes. The user guides the method by drawing colored strokes, giving, for each set, some pixels belonging to it. If the majority of region-based methods extract a single object from the background, some algorithms, proposed during the last decade, are able to solve multi-class interactive segmentation problems, i.e., to extract more than two sets of pixels. The main contribution of this work is the design of a new multi-class interactive segmentation method. This algorithm is based on the minimization of a cost function that can be represented by a factor graph. It integrates a supervised learning classification method checking that the produced segmentation is consistent with the indications given by the user, a new regularization term, and a preprocessing step grouping pixels into small homogeneous regions called superpixels. The use of an over-segmentation method to produce these superpixels is a key step in the proposed interactive segmentation method : it significantly reduces the computational complexity and handles the segmentation of images containing several millions of pixels, by keeping the execution time small enough to ensure comfortable use of the method. The second contribution of our work is an evaluation of over-segmentation algorithms. We provide a new dataset, with images of different sizes with a majority of big images. This review has also allowed us to design a new over-segmentation algorithm and to evaluate it.

This work describes design and implementation of a tool for creating photomontages. The tool is based on methods of semi-automatic image segmentation. Work outlines problems of segmentation of image data and benefits of interaction with the user. It analyzes different approaches to interactive image segmentation, explains their principles and shows their positive and negative aspects. It also presents advantages and disadvantages of currently used photo-editing applications. Proposes application for creating photomontages which consists of two steps: Extraction of an object from picture and insertion of it into another picture. The first step uses the method of semi-automatic segmentation GrabCut based on the graph theory. The work also includes comparison between application and other applications in which it is possible to create a photomontage, and application tests done by users.

This thesis investigates two well defined problems in image segmentation, viz. interactive and semantic image segmentation. Interactive segmentation involves power assisting a user in cutting out objects from an image, whereas semantic segmentation involves partitioning pixels in an image into object categories. We investigate various models and energy formulations for both these problems in this thesis. In order to improve the performance of interactive systems, low level texture features are introduced as a replacement for the more commonly used RGB features. To quantify the improvement obtained by using these texture features, two annotated datasets of images are introduced (one consisting of natural images, and the other consisting of camouflaged objects). A significant improvement in performance is observed when using texture features for the case of monochrome images and images containing camouflaged objects. We also explore adding mid-level cues such as shape constraints into interactive segmentation by introducing the idea of geodesic star convexity, which extends the existing notion of a star convexity prior in two important ways: (i) It allows for multiple star centres as opposed to single stars in the original prior and (ii) It generalises the shape constraint by allowing for Geodesic paths as opposed to Euclidean rays. Global minima of our energy function can be obtained subject to these new constraints. We also introduce Geodesic Forests, which exploit the structure of shortest paths in implementing the extended constraints. These extensions to star convexity allow us to use such constraints in a practical segmentation system. This system is evaluated by means of a “robot user” to measure the amount of interaction required in a precise way, and it is shown that having shape constraints reduces user effort significantly compared to existing interactive systems. We also introduce a new and harder dataset which augments the existing GrabCut dataset with more realistic images and ground truth taken from the PASCAL VOC segmentation challenge. In the latter part of the thesis, we bring in object category level information in order to make the interactive segmentation tasks easier, and move towards fully automated semantic segmentation. An algorithm to automatically segment humans from cluttered images given their bounding boxes is presented. A top down segmentation of the human is obtained using classifiers trained to predict segmentation masks from local HOG descriptors. These masks are then combined with bottom up image information in a local GrabCut like procedure. This algorithm is later completely automated to segment humans without requiring a bounding box, and is quantitatively compared with other semantic segmentation methods. We also introduce a novel way to acquire large quantities of segmented training data relatively effortlessly using the Kinect. In the final part of this work, we explore various semantic segmentation methods based on learning using bottom up super-pixelisations. Different methods of combining multiple super-pixelisations are discussed and quantitatively evaluated on two segmentation datasets. We observe that simple combinations of independently trained classifiers on single super-pixelisations perform almost as good as complex methods based on jointly learning across multiple super-pixelisations. We also explore CRF based formulations for semantic segmentation, and introduce novel visual words based object boundary description in the energy formulation. The object appearance and boundary parameters are trained jointly using structured output learning methods, and the benefit of adding pairwise terms is quantified on two different datasets.

Submitted by Rosivalda Pereira (mrs.pereira@ufma.br) on 2017-08-16T21:13:39Z No. of bitstreams: 1 HeberSousa.pdf: 2248069 bytes, checksum: e89eac1d4562ac1f2f53007d699f9c71 (MD5) / Made available in DSpace on 2017-08-16T21:13:39Z (GMT). No. of bitstreams: 1 HeberSousa.pdf: 2248069 bytes, checksum: e89eac1d4562ac1f2f53007d699f9c71 (MD5) Previous issue date: 2013-01-29 / Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) / Lung cancer is one of the most common of malignant tumors. It also has one of the highest rates of mortality among cancers. The reason for this is mainly linked to late diagnosis of the disease. For early detection of disease is very helpful to use medical images as support, the most important being, CT. With the acquisition of digital images is becoming more common to use computer systems for medical imaging. These systems assist in the clinical diagnosis, disease monitoring, and in some cases is used as a support for surgery. Because the search for new ways of human-computer interaction, natural interaction arises, which aims to provide a form of control with higher cognition. This control is usually performed using gestures. Interactions of gestures can be useful in controlling medical imaging systems and can ensure necessary sterility in operating rooms, because they are not required contacts manuals. Among the activities computer assisted important for the treatment of lung cancer, we have the segmentation of nodules. The segmentation of nodules can be performed automatically, semiautomatically or interactively. It is useful to speed up the diagnostic process, taking measurements, or observe the morphological appearance of the nodule. The objective of this study is to investigate the use of natural interaction interface for activities such as medical image visualization and segmentation of pulmonary nodules. The paper proposes the study of interaction techniques based on gestures to segment nodules in an interactive and semiautomatic. Finally, conducting experiments to evaluate the techniques proposed in the items ease of use, intuitiveness, accuracy and comfortability / O câncer de pulmão é um dos mais comuns dentre os tumores malignos. Ele também possui uma das taxas mais altas de mortalidade dentre os tipos de câncer. O motivo disso está ligado principalmente ao diagnóstico tardio da doença. Para a sua detecção precoce é muito útil a utilização de imagens médicas como apoio, sendo a mais importante, a tomografia computadorizada. Com a aquisição digital das imagens está cada vez mais comum a utilização de sistemas computacionais de visualização médica. Estes sistemas auxiliam no diagnóstico clínico, no acompanhamento de doenças, e em alguns casos é utilizado como apoio a cirurgias. Em virtude da busca por novos meios de interação humano-computador, surge a interação natural, que objetiva uma forma de controle mais próximo cognitivamente das ações realizadas, e geralmente é realizada através de gestos. Interações por gestos podem ser úteis no controle de sistemas de visualização médica e podem garantir a esterilização necessária em salas cirúrgicas, pois não são necessários contatos manuais. Dentre as atividades assistidas por computador importantes para o tratamento do câncer pulmonar, temos a segmentação de nódulos. A segmentação de nódulos pode ser realizada de forma automática, semiautomática ou interativamente. Elas são úteis para agilizar o processo de diagnóstico, realizar medições, ou observar o aspecto morfológico do nódulo. O objetivo do presente trabalho é investigar a utilização da interação natural como interface para atividades de visualização de imagens médicas e segmentação de nódulos pulmonares. Foi implementada uma série de ferramentas de segmentação, interativas e semiautomáticas, controladas a partir de gestos. Estes gestos foram desenvolvidos a partir de imagens capturadas por uma câmera especial chamada Kinect, que traduz a imagem em mapas de profundidade, podendo medir com precisão a distância de objetos na cena. Ao final do estudo, foi realizado experimentos para avaliar as técnicas propostas nos quesitos facilidade de uso, intuitividade, conforto e precisão.

As we navigate the world, for example when driving a car from our home to the work place, we continuously perceive the 3D structure of our surroundings and intuitively recognise the objects we see. Such capabilities help us in our everyday lives and enable free and accurate movement even in completely unfamiliar places. We largely take these abilities for granted, but for robots, the task of understanding large outdoor scenes remains extremely challenging. In this thesis, I develop novel algorithms for (near) real-time dense 3D reconstruction and semantic segmentation of large-scale outdoor scenes from passive cameras. Motivated by "smart glasses" for partially sighted users, I show how such modeling can be integrated into an interactive augmented reality system which puts the user in the loop and allows her to physically interact with the world to learn personalized semantically segmented dense 3D models. In the next part, I show how sparse but very accurate 3D measurements can be incorporated directly into the dense depth estimation process and propose a probabilistic model for incremental dense scene reconstruction. To relax the assumption of a stereo camera, I address dense 3D reconstruction in its monocular form and show how the local model can be improved by joint optimization over depth and pose. The world around us is not stationary. However, reconstructing dynamically moving and potentially non-rigidly deforming texture-less objects typically require "contour correspondences" for shape-from-silhouettes. Hence, I propose a video segmentation model which encodes a single object instance as a closed curve, maintains correspondences across time and provide very accurate segmentation close to object boundaries. Finally, instead of evaluating the performance in an isolated setup (IoU scores) which does not measure the impact on decision-making, I show how semantic 3D reconstruction can be incorporated into standard Deep Q-learning to improve decision-making of agents navigating complex 3D environments.

La thèse porte sur la segmentation spatio-temporelle et interactive d'images cardiaquesdynamiques. Elle s'inscrit dans le projet ANR 3DSTRAIN du programme"Technologies pour la Santé et l'Autonomie" qui a pour objectif d'estimer de façoncomplète, dense et sur plusieurs modalités d'imagerie 3D+t (telles que l'imageriepar résonance magnétique (IRM), la tomographie par émission monophotonique(TEMP) et l'échocardiographie) l'indice de déformation du muscle cardiaque : lestrain. L'estimation du strain nécessite une étape de segmentation qui doit être laplus précise possible pour fournir une bonne évaluation de cet indice. Nos travauxse sont orientés sur deux axes principaux : (1) le développement d'un modèle desegmentation conforme à la morphologie du muscle cardiaque et (2) la possibilitéde corriger interactivement et intuitivement le résultat de la segmentation obtenuegrâce à ce modèle. / This thesis focuses on the spatio-temporal and interactive segmentation of dynamiccardiac images. It is a part of the ANR 3DSTRAIN project of program "Technologiesfor Health and Autonomy" which aims to estimate full, dense and on several3D+t imaging modalities (such as Magnetic Resonance Imaging (MRI), Single PhotonEmission Computed Tomography (SPECT) and echocardiography) the indexof deformation of the heart muscle : the strain. The strain estimation requires asegmentation step which must be as precise as possible to provide a good estimationof this index. Our work was focused on two main areas : (1) the development of asegmentation model conforms to the shape of the heart muscle and (2) the abilityto interactively and intuitively correct the segmentation's result obtained with thismodel.

A segmentação de imagens é um problema muito importante em visão computacional, no qual uma imagem é dividida em regiões relevantes, tal como para isolar objetos de interesse de uma dada aplicação. Métodos de segmentação baseados na transformada imagem-floresta (IFT, Image Foresting Transform), com funções de conexidade monotonicamente incrementais (MI) têm alcançado um grande sucesso em vários contextos. Na segmentação interativa de imagens, na qual o usuário pode especificar o objeto desejado, novas sementes podem ser adicionadas e/ou removidas para corrigir a rotulação até conseguir a segmentação esperada. Este processo gera uma sequência de IFTs que podem ser calculadas de modo mais eficiente pela DIFT (Differential Image Foresting Transform). Recentemente, funções de conexidade não monotonicamente incrementais (NMI) têm sido usadas com sucesso no arcabouço da IFT no contexto de segmentação de imagens, permitindo incorporar informações de alto nível, tais como, restrições de forma, polaridade de borda e restrição de conexidade, a fim de customizar a segmentação para um dado objeto desejado. Funções não monotonicamente incrementais foram também exploradas com sucesso na geração de superpixels, via sequências de execuções da IFT. Neste trabalho, apresentamos um estudo sobre a Transformada Imagem-Floresta Diferencial no caso de funções NMI. Nossos estudos indicam que o algoritmo da DIFT original apresenta uma série de inconsistências para funções não monotonicamente incrementais. Este trabalho estende a DIFT, visando incorporar um subconjunto das funções NMI em grafos dirigidos e mostrar sua aplicação no contexto da geração de superpixels. Outra aplicação que é apresentada para difundir a relevância das funções NMI é o algoritmo Bandeirantes para perseguição de bordas e rastreamento de curvas. / Image segmentation is a problem of great relevance in computer vision, in which an image is divided into relevant regions, such as to isolate an object of interest for a given application. Segmentation methods with monotonically incremental connectivity functions (MI) based on the Image Foresting Transform (IFT) have achieved great success in several contexts. In interactive segmentation of images, in which the user is allowed to specify the desired object, new seeds can be added and/or removed to correct the labeling until achieving the expected segmentation. This process generates a sequence of IFTs that can be calculated more efficiently by the Differential Image Foresting Trans- form (DIFT). Recently, non-monotonically incremental connectivity functions (NMI) have been used successfully in the IFT framework in the context of image segmentation, allowing the incorporation of shape, boundary polarity, and connectivity constraints, in order to customize the segmentation for a given target object. Non-monotonically incremental functions were also successfully exploited in the generation of superpixels, via sequences of IFT executions. In this work, we present a study of the Differential Image Foresting Transform in the case of NMI functions. Our research indicates that the original DIFT algorithm presents a series of inconsistencies for non-monotonically incremental functions. This work extends the DIFT algorithm to NMI functions in directed graphs, and shows its application in the context of the generation of superpixels. Another application that is presented to spread the relevance of NMI functions is the Bandeirantes algorithm for curve tracing and boundary tracking.