This thesis is devoted to the problems of defining and developing the basic building blocks of an automated surveillance system. As its initial step, a background-modeling algorithm is described for segmenting moving objects from the background, which is capable of adapting to dynamic scene conditions, as well as determining shadows of the moving objects. After obtaining binary silhouettes for targets, object association between consecutive frames is achieved by a hypothesis-based tracking method. Both of these tasks provide basic information for higher-level processing, such as activity analysis and object identification. In order to recognize the nature of an event occurring in a scene, hidden Markov models (HMM) are utilized. For this aim, object trajectories, which are obtained through a successful track, are written as a sequence of flow vectors that capture the details of instantaneous velocity and location information. HMMs are trained with sequences obtained from usual motion patterns and abnormality is detected by measuring the distance to these models. Finally, MPEG-7 visual descriptors are utilized in a regional manner for object identification. Color structure and homogeneous texture parameters of the independently moving objects are extracted and classifiers, such as Support Vector Machine (SVM) and Bayesian plug-in (Mahalanobis distance), are utilized to test the performance of the proposed person identification mechanism. The simulation results with all the above building blocks give promising results, indicating the possibility of constructing a fully automated surveillance system for the future.

Surveillance networks are typically monitored by a few people, viewing several monitors displaying the camera feeds. It is then very dicult for a human op- erator to eectively detect events as they happen. Recently, computer vision research has begun to address ways to automatically process some of this data, to assist human operators. Object tracking, event recognition, crowd analysis and human identication at a distance are being pursued as a means to aid human operators and improve the security of areas such as transport hubs. The task of object tracking is key to the eective use of more advanced technolo- gies. To recognize an event people and objects must be tracked. Tracking also enhances the performance of tasks such as crowd analysis or human identication. Before an object can be tracked, it must be detected. Motion segmentation tech- niques, widely employed in tracking systems, produce a binary image in which objects can be located. However, these techniques are prone to errors caused by shadows and lighting changes. Detection routines often fail, either due to erro- neous motion caused by noise and lighting eects, or due to the detection routines being unable to split occluded regions into their component objects. Particle l- ters can be used as a self contained tracking system, and make it unnecessary for the task of detection to be carried out separately except for an initial (of- ten manual) detection to initialise the lter. Particle lters use one or more extracted features to evaluate the likelihood of an object existing at a given point each frame. Such systems however do not easily allow for multiple objects to be tracked robustly, and do not explicitly maintain the identity of tracked objects. This dissertation investigates improvements to the performance of object tracking algorithms through improved motion segmentation and the use of a particle lter. A novel hybrid motion segmentation / optical ow algorithm, capable of simulta- neously extracting multiple layers of foreground and optical ow in surveillance video frames is proposed. The algorithm is shown to perform well in the presence of adverse lighting conditions, and the optical ow is capable of extracting a mov- ing object. The proposed algorithm is integrated within a tracking system and evaluated using the ETISEO (Evaluation du Traitement et de lInterpretation de Sequences vidEO - Evaluation for video understanding) database, and signi- cant improvement in detection and tracking performance is demonstrated when compared to a baseline system. A Scalable Condensation Filter (SCF), a particle lter designed to work within an existing tracking system, is also developed. The creation and deletion of modes and maintenance of identity is handled by the underlying tracking system; and the tracking system is able to benet from the improved performance in uncertain conditions arising from occlusion and noise provided by a particle lter. The system is evaluated using the ETISEO database. The dissertation then investigates fusion schemes for multi-spectral tracking sys- tems. Four fusion schemes for combining a thermal and visual colour modality are evaluated using the OTCBVS (Object Tracking and Classication in and Beyond the Visible Spectrum) database. It is shown that a middle fusion scheme yields the best results and demonstrates a signicant improvement in performance when compared to a system using either mode individually. Findings from the thesis contribute to improve the performance of semi- automated video processing and therefore improve security in areas under surveil- lance.

A localização de objetos em uma imagem e acompanhamento de seu deslocamento numa sequência de imagens são tarefas de interesse teórico e prático. Aplicações de reconhecimento e rastreamento de padrões e objetos tem se difundido ultimamente, principalmente no ramo de controle, automação e vigilância. Esta dissertação apresenta um método eficaz para localizar e rastrear automaticamente objetos em vídeos. Para tanto, foi utilizado o caso do rastreamento da bola em vídeos esportivos, especificamente o jogo de futebol. O algoritmo primeiramente localiza a bola utilizando segmentação, eliminação e ponderação de candidatos, seguido do algoritmo de Viterbi, que decide qual desses candidatos representa efetivamente a bola. Depois de encontrada, a bola é rastreada utilizando o Filtro de Partículas auxiliado pelo método de semelhança de histogramas. Não é necessária inicialização da bola ou intervenção humana durante o algoritmo. Por fim, é feita uma comparação do Filtro de Kalman com o Filtro de Partículas no escopo do rastreamento da bola em vídeos de futebol. E, adicionalmente, é feita a comparação entre as funções de semelhança para serem utilizadas no Filtro de Partículas para o rastreamento da bola. Dificuldades, como a presença de ruído e de oclusão, tanto parcial como total, tiveram de ser contornadas. / The location of objects in an image and tracking its movement in a sequence of images is a task of theoretical and practical interest. Applications for recognition and tracking of patterns and objects have been spread lately, especially in the field of control, automation and vigilance. This dissertation presents an effective method to automatically locate and track objects in videos. Thereto, we used the case of tracking the ball in sports videos, specifically the game of football. The algorithm first locates the ball using segmentation, elimination and the weighting of candidates, followed by a Viterbi algorithm, which decides which of these candidates is actually the ball. Once found, the ball is tracked using the Particle Filter aided by the method of similarity of histograms. It is not necessary to initialize the ball or any human intervention during the algorithm. Next, a comparison of the Kalman Filter to Particle Filter in the scope of tracking the ball in soccer videos is made. And in addition, a comparison is made between the functions of similarity to be used in the Particle Filter for tracking the ball. Difficulties, such as the presence of noise and occlusion, in part or in total, had to be circumvented.

Applying machine learning techniques for real-time detection and tracking of objects in very high-resolution video is a problem that has not been extensively studied. In this thesis, the practical uses of object detection for airport remote towers are explored. We present a Kalman filter-based tracking framework for low-latency aircraft tracking in very high-resolution video streams. The object detector was trained and tested on a dataset containing 3000 labelled images of aircrafts taken at Swedish airports, reaching an mAP of 90.91% with an average IoU of 89.05% on the test set. The tracker was benchmarked on remote tower video footage from Örnsköldsvik and Sundsvall using slightly modified variants of the MOT-CLEAR and ID metrics for multiple object trackers, obtaining an IDF1 score of 91.9%, and a MOTA score of 83.3%. The prototype runs the tracking pipeline on seven high resolution cameras simultaneously at 10 Hz on a single thread, suggesting large potential speed gains being attainable through parallelization.

Ces dernières années se caractérisent par la prolifération des systèmes de vidéo-surveillance et par l’automatisation des traitements que ceux-ci intègrent. Parallèlement, le problème du suivi d’objets est devenu en quelques années un problème récurrent dans de nombreux domaines et notamment en vidéo-surveillance. Dans le cadre de cette thèse, nous proposons une nouvelle méthode de suivi d’objet, basée sur la méthode Ensemble Tracking et intégrant deux améliorations majeures. La première repose sur une séparation de l’espace hétérogène des caractéristiques en un ensemble de sous-espaces homogènes appelés modules et sur l’application, sur chacun d’eux, d’un algorithme basé Ensemble Tracking. La seconde adresse, quant à elle, l’apport d’une solution à la nouvelle problématique de suivi induite par cette séparation des espaces, à savoir la construction d’un filtre particulaire spécifique exploitant une pondération des différents modules utilisés afin d’estimer à la fois, pour chaque image de la séquence, la position et les dimensions de l’objet suivi, ainsi que la combinaison linéaire des différentes décisions modulaires conduisant à l’observation la plus discriminante. Les différents résultats que nous présentons illustrent le bon fonctionnement global et individuel de l’ensemble des propriétés spécifiques de la méthode et permettent de comparer son efficacité à celle de plusieurs algorithmes de suivi de référence. De plus, l’ensemble des travaux a fait l’objet d’un développement industriel sur les consoles de traitement de la société partenaire. En conclusion de ces travaux, nous présentons les perspectives que laissent entrevoir ces développements originaux, notamment en exploitant les possibilités offertes par la modularité de l’algorithme ou encore en rendant dynamique le choix des modules utilisés en fonction de l’efficacité de chacun dans une situation donnée. / Recent years have been characterized by the overgrowth of video-surveillance systems and by automation of treatments they integrate. At the same time, object tracking has become, within years, a recurring problem in many domains and particularly in video-surveillance. In this dissertation, we propose a new object tracking method, based on the Ensemble Tracking method and integrating two main improvements. The first one lies on the separation of the heterogeneous feature space into a set of homogenous sub-spaces called modules and on the application, on each of them, of an Ensemble Tracking-based algorithm. The second one deals with the new tracking problem induced by this separation by building a specific particle filter. This filter weights each used module in order to estimate, for each frame in the sequence, both position and dimensions of the tracked object and the linear combination of modular decisions leading to the most discriminative observation. The results we present illustrate the global and individual efficiency of all the specific properties of our method and allow comparing this efficiency with the one of several reference tracking algorithms. Furthermore, all this work has led to an industrial development on the treatment systems of the partner company. In conclusion of this work, we present the prospects generated by these original developments, more particularly using the possibilities offered by the algorith mmodularity or making the modules choice dynamic according to their efficiency in a given situation.

A localização de objetos em uma imagem e acompanhamento de seu deslocamento numa sequência de imagens são tarefas de interesse teórico e prático. Aplicações de reconhecimento e rastreamento de padrões e objetos tem se difundido ultimamente, principalmente no ramo de controle, automação e vigilância. Esta dissertação apresenta um método eficaz para localizar e rastrear automaticamente objetos em vídeos. Para tanto, foi utilizado o caso do rastreamento da bola em vídeos esportivos, especificamente o jogo de futebol. O algoritmo primeiramente localiza a bola utilizando segmentação, eliminação e ponderação de candidatos, seguido do algoritmo de Viterbi, que decide qual desses candidatos representa efetivamente a bola. Depois de encontrada, a bola é rastreada utilizando o Filtro de Partículas auxiliado pelo método de semelhança de histogramas. Não é necessária inicialização da bola ou intervenção humana durante o algoritmo. Por fim, é feita uma comparação do Filtro de Kalman com o Filtro de Partículas no escopo do rastreamento da bola em vídeos de futebol. E, adicionalmente, é feita a comparação entre as funções de semelhança para serem utilizadas no Filtro de Partículas para o rastreamento da bola. Dificuldades, como a presença de ruído e de oclusão, tanto parcial como total, tiveram de ser contornadas. / The location of objects in an image and tracking its movement in a sequence of images is a task of theoretical and practical interest. Applications for recognition and tracking of patterns and objects have been spread lately, especially in the field of control, automation and vigilance. This dissertation presents an effective method to automatically locate and track objects in videos. Thereto, we used the case of tracking the ball in sports videos, specifically the game of football. The algorithm first locates the ball using segmentation, elimination and the weighting of candidates, followed by a Viterbi algorithm, which decides which of these candidates is actually the ball. Once found, the ball is tracked using the Particle Filter aided by the method of similarity of histograms. It is not necessary to initialize the ball or any human intervention during the algorithm. Next, a comparison of the Kalman Filter to Particle Filter in the scope of tracking the ball in soccer videos is made. And in addition, a comparison is made between the functions of similarity to be used in the Particle Filter for tracking the ball. Difficulties, such as the presence of noise and occlusion, in part or in total, had to be circumvented.

In this thesis we propose a novel approach for tracking multiple objects using structural information. The objects are tracked by combining particle filter and frame description with Attributed Relational Graphs (ARGs). We start by learning a structural probabilistic model graph from annotated images. The graphs are then used to evaluate the current tracking state and to correct it, if necessary. By doing so, the proposed method is able to deal with challenging situations such as abrupt motion and tracking loss due to occlusion. The main contribution of this thesis is the exploration of the learned probabilistic structural model. By using it, the structural information of the scene itself is used to guide the object detection process in case of tracking loss. This approach differs from previous works, that use structural information only to evaluate the scene, but do not consider it to generate new tracking hypotheses. The proposed approach is very flexible and it can be applied to any situation in which it is possible to find structural relation patterns between the objects. Object tracking may be used in many practical applications, such as surveillance, activity analysis or autonomous navigation. In this thesis, we explore it to track multiple objects in sports videos, where the rules of the game create some structural patterns between the objects. Besides detecting the objects, the tracking results are also used as an input for recognizing the action each player is performing. This step is performed by classifying a segment of the tracking sequence using Hidden Markov Models (HMMs). The proposed tracking method is tested on several videos of table tennis matches and on the ACASVA dataset, showing that the method is able to continue tracking the objects even after occlusion or when there is a camera cut. / Nesta tese, uma nova abordagem para o rastreamento de múltiplos objetos com o uso de informação estrutural é proposta. Os objetos são rastreados usando uma combinação de filtro de partículas com descrição das imagens por meio de Grafos Relacionais com Atributos (ARGs). O processo é iniciado a partir do aprendizado de um modelo de grafo estrutural probabilístico utilizando imagens anotadas. Os grafos são usados para avaliar o estado atual do rastreamento e corrigi-lo, se necessário. Desta forma, o método proposto é capaz de lidar com situações desafiadoras como movimento abrupto e perda de rastreamento devido à oclusão. A principal contribuição desta tese é a exploração do modelo estrutural aprendido. Por meio dele, a própria informação estrutural da cena é usada para guiar o processo de detecção em caso de perda do objeto. Tal abordagem difere de trabalhos anteriores, que utilizam informação estrutural apenas para avaliar o estado da cena, mas não a consideram para gerar novas hipóteses de rastreamento. A abordagem proposta é bastante flexível e pode ser aplicada em qualquer situação em que seja possível encontrar padrões de relações estruturais entre os objetos. O rastreamento de objetos pode ser utilizado para diversas aplicações práticas, tais como vigilância, análise de atividades ou navegação autônoma. Nesta tese, ele é explorado para rastrear diversos objetos em vídeos de esporte, na qual as regras do jogo criam alguns padrões estruturais entre os objetos. Além de detectar os objetos, os resultados de rastreamento também são usados como entrada para reconhecer a ação que cada jogador está realizando. Esta etapa é executada classificando um segmento da sequência de rastreamento por meio de Modelos Ocultos de Markov (HMMs). A abordagem de rastreamento proposta é testada em diversos vídeos de jogos de tênis de mesa e na base de dados ACASVA, demonstrando a capacidade do método de lidar com situações de oclusão ou cortes de câmera.

Submitted by Renata Lopes (renatasil82@gmail.com) on 2018-11-23T09:57:40Z No. of bitstreams: 1 douglascoelhobragadeoliveira.pdf: 19144398 bytes, checksum: 652398b01779c3899281a6ba454c143a (MD5) / Approved for entry into archive by Adriana Oliveira (adriana.oliveira@ufjf.edu.br) on 2018-11-23T12:48:28Z (GMT) No. of bitstreams: 1 douglascoelhobragadeoliveira.pdf: 19144398 bytes, checksum: 652398b01779c3899281a6ba454c143a (MD5) / Made available in DSpace on 2018-11-23T12:48:28Z (GMT). No. of bitstreams: 1 douglascoelhobragadeoliveira.pdf: 19144398 bytes, checksum: 652398b01779c3899281a6ba454c143a (MD5) Previous issue date: 2018-09-19 / CAPES - Coordenação de Aperfeiçoamento de Pessoal de Nível Superior / Realidade Aumentada (RA) é uma tecnologia que permite combinar objetos virtuais tridimensionais com um ambiente predominantemente real, de forma a construir um novo ambiente onde os objetos reais e virtuais podem interagir uns com os outros em tempo real. Para fazer isso, é necessário encontrar a pose do observador (câmera, HMD, óculos inteligentes, etc.) em relação a um sistema de coordenadas global. Geralmente, algum objeto físico conhecido é usado para marcar o referencial para as projeções e para a posição do observador. O problema de Localização e Mapeamento Simultâneo (SLAM) se origina da comunidade de robótica como uma condição necessária para se construir robôs verdadeiramente autônomos, capazes de se auto localizarem em um ambiente desconhecido ao mesmo tempo que constroem um mapa da cena observada a partir de informações capturadas por um conjunto de sensores. A principal contribuição do SLAM para a RA é permitir aplicações em ambientes despreparados, ou seja, sem marcadores. No entanto, ao eliminar o marcador, perdemos o referencial para a projeção dos objetos virtuais e a principal fonte de interação entre os elementos reais e virtuais. Embora o mapa gerado possa ser processado a fim de encontrar uma estrutura conhecida, como um plano predominante, para usá-la como referencial, isso ainda não resolve a questão das interações. Na literatura recente, encontramos trabalhos que integram um sistema de reconhecimento de objetos ao SLAM e incorporam tais objetos ao mapa. Frequentemente, assume-se um mapa estático, devido às limitações das técnicas envolvidas, de modo que o objeto é usado apenas para fornecer informações semânticas sobre a cena. Neste trabalho, propomos um novo framework que permite estimar simultaneamente a posição da câmera e de objetos para cada quadro de vídeo em tempo real. Dessa forma, cada objeto é independente e pode se mover pelo mapa livremente, assim como nos métodos baseados em marcadores, mas mantendo as vantagens que o SLAM fornece. Implementamos a estrutura proposta sobre um sistema SLAM de última geração a fim de validar nossa proposta e demonstrar a potencial aplicação em Realidade Aumentada. / Augmented Reality (AR) is a technology that allows combining three-dimensional virtual objects with an environment predominantly real in a way to build a new environment where both real and virtual objects can interact with each other in real-time. To do this, it is required to nd the pose of the observer (camera, HMD, smart glasses etc) in relation to a global coordinate system. Commonly, some well known physical object, called marker, is used to de ne the referential for both virtual objects and the observer's position. The Simultaneous Localization and Mapping (SLAM) problem borns from robotics community as a way to build truly autonomous robots by allowing they to localize themselves while they build a map of the observed scene from the input data of their coupled sensors. SLAM-based Augmented Reality is an active and evolving research line. The main contribution of the SLAM to the AR is to allow applications on unprepared environments, i.e., without markers. However, by eliminating the marker object, we lose the referential for virtual object projection and the main source of interaction between real and virtual elements. Although the generated map can be processed in order to nd a known structure, e.g. a predominant plane, to use it as the referential system, this still not solve for interactions. In the recent literature, we can found works that integrate an object recognition system to the SLAM in a way the objects are incorporated into the map. The SLAM map is frequently assumed to be static, due to limitations on techniques involved, so that on these works the object is just used to provide semantic information about the scene. In this work, we propose a new framework that allows estimating simultaneously the camera and object positioning for each camera image in real time. In this way, each object is independent and can move through the map as well as in the marker-based methods but with the SLAM advantages kept. We develop our proposed framework over a stateof- the-art SLAM system in order to evaluate our proposal and demonstrate potentials application in Augmented Reality.

Humans live in physical world and perform activities that are physical, natural and biological. But humans are forced to shift explicitly from physical world to virtual world and vice-versa in performing computer aided physical activities. The research reported here is investigating: How implicit human-computer interaction can be used as a means to bridge the gap between physical world and virtual world. An agent-centric approach is introduced to extend ubiquitous computing to unlimited geographical space and a framework for implicit human-computer interaction is also discussed. The benefits of standardized ontologies are used as a base upon which this framework is built. This semantic approach together with agent-centric approach is discussed to visualize the visions of implicit Human-Computer Interaction (i-HCI). / PHYVIR project

In many applications, the dataset for classification may be highly imbalanced where most of the instances in the training set may belong to some of the classes (majority classes), while only a few instances are from the other classes (minority classes). Conventional classifiers will strongly favor the majority class and ignore the minority instances. The imbalance problem can occur in both binary data classification and also in ordinal regression. Ordinal regression is a supervised approach for learning the ordinal relationship between classes. Extensive research has been performed for addressing imbalanced datasets for binary classification; however, current methods do not address within-class imbalance and between-class imbalance at the same time. Similarly, there has been very little research work on addressing imbalanced datasets for ordinal regression. Although current standard oversampling methods can be used to improve the dataset class distribution, they do not consider the ordinal relationship between the classes. The class imbalance problem is a big challenge in classification problems. Most of the clinical datasets are highly imbalanced, which can weaken the performance of classifiers significantly. In this research, the imbalanced dataset classification problem is also examined in the context of a clinical application, particularly pelvic organ prolapse diagnosis. Pelvic organ prolapse (POP) is a major health problem that affects between 30-50% of women in the U.S. Although clinical examination is currently used to diagnose POP, there is still little evidence on specific risk factors that are directly related to particular types of POP and their severity or stages (Stage 0-IV). Data from dynamic MRI related to the movement of pelvic organs has the potential to improve POP prediction but it is currently analyzed manually limiting its exploration and use to small datasets. Moreover, POP is a disorder with multiple stages that are ordinal and whose distribution is highly imbalanced. The main goal of this research is two-fold. The first goal is to design new oversampling methods for imbalanced datasets for both binary classification and ordinal regression. The second goal is to automatically track, segment, and classify the trajectory of multiple organs on dynamic MRI to quantitatively describe pelvic organ movement. The extracted image-based data along with the designed oversampling methods will be used to improve the diagnosis of POP. The proposed research consists of three major objectives: 1) to design a new oversampling technique for binary imbalanced dataset classification; 2) to design a novel oversampling technique for ordinal regression with imbalanced datasets; and 3) to design a two-stage method to automatically track and segment multiple pelvic organs on dynamic MRI for improving the prediction of multi-stage POP with imbalanced datasets. The proposed research aims to provide robust oversampling techniques and image processing models that can (1) effectively handle highly imbalanced datasets for both binary classification and ordinal regression, and (2) automatically track and segment multiple deformable structures for feature extraction from low contrast and nonhomogeneous images and classify them using the resulted trajectories. This research will set the foundation towards a computer-aided decision support system that can automatically extract and analyze image and clinical data to improve the prediction of disorders where the dataset is highly imbalanced through personalized and evidence-based assessment.