A method for real time tracking of non-rigid arbitrary objects is proposed in this study. The approach builds on and extends work on multidimensional color histogram based target representation, which is enhanced by spatial masking with a monotonically decreasing kernel profile prior to back-projection. The masking suppresses the influence of the background pixels and induces a spatially smooth target model representation suitable for gradient-based optimization. The main idea behind this approach is that an increase in the number of quantized feature spaces used to generate the target probability distribuition function during histogram back-projection can lead to improved target localization. Target localization is performed using the recursive Mean shift procedure, which climbs the underlying density graidients of the discrete data to find the mode (peak) of the distribution. Finally, the real time test cases, such as occlusion, target scale and orientation changes, varying illumination and background clutter, are demonstrated.

An improved object tracking algorithm based Kalman filtering is developed in this thesis. The algorithm uses a median filter and morphological operations during tracking. The problem created by object shadows is identified and the primary focus is to incorporate shadow detection and removal to improve tracking multiple objects in complex scenes. It is shown that the Kalman filter, without the improvements, fails to remove shadows that connect different objects. The application of the median filter helps the separation of different objects and thus enables the tracking of multiple objects individually. The performances of the Kalman filter and the improved tracking algorithm were tested on a highway video sequence of moving cars and it is shown that the proposed algorithm yields better performance in the presence of shadows.

O rastreamento de objetos em vídeos representa um importante problema na área de processamento de imagens, quer seja pelo grande número de aplicações envolvidas, ou pelo grau de complexidade que pode ser apresentado. Como exemplo de aplicações, podemos citar sua utilização em áreas como robótica móvel, interface homem-máquina, medicina, automação de processo industriais até aplicações mais tracionais como vigilância e monitoramento de trafego. O aumento na complexidade do rastreamento se deve principalmente a interação do objeto rastreado com outros elementos da cena do vídeo, especialmente nos casos de oclusões parciais ou totais. Quando uma oclusão ocorre a informação sobre a localização do objeto durante o rastreamento é perdida parcial ou totalmente. Métodos de filtragem estocástica, utilizados para o rastreamento de objetos, como os Filtros de Partículas não apresentam resultados satisfatórios na presença de oclusões totais, onde temos uma descontinuidade na trajetória do objeto. Portanto torna-se necessário o desenvolvimento de métodos específicos para tratar o problema de oclusão total. Nesse trabalho, nós desenvolvemos uma abordagem para tratar o problema de oclusão total no rastreamento de objetos utilizando Filtro de Partículas baseados em Monte Carlo via Cadeia de Markov (MCCM) com função geradora de partículas adaptativa. Durante o rastreamento do objeto, em situações onde não há oclusões, nós utilizamos uma função de probabilidade geradora simétrica. Entretanto, quando uma oclusão total, ou seja, uma descontinuidade na trajetória é detectada, a função geradora torna-se assimétrica, criando um termo de “inércia” ou “arraste” na direção do deslocamento do objeto. Ao sair da oclusão, o objeto é novamente encontrado e a função geradora volta a ser simétrica novamente. / The object tracking on video is an important task in image processing area either for the great number of involved applications, or for the degree of complexity that can be presented. How example of application, we can cite its use from robotic area, machine-man interface, medicine, automation of industry process to vigilance and traffic control applications. The increase of complexity of tracking is occasioned principally by interaction of tracking object with other objects on video, specially when total or partial occlusions occurs. When a occlusion occur the information about the localization of tracking object is lost partially or totally. Stochastic filtering methods, like Particle Filter do not have satisfactory results in the presence of total occlusions. Total occlusion can be understood like discontinuity in the object trajectory. Therefore is necessary to develop specific method to handle the total occlusion task. In this work, we develop an approach to handle the total occlusion task using MCMC-Particle Filter with adaptive sampling probability function. When there is not occlusions we use a symmetric probability function to sample the particles. However, when there is a total occlusion, a discontinuity in the trajectory is detected, and the probability sampling function becomes asymmetric. This break of symmetry creates a “drift” or “inertial” term in object shift direction. When the tracking object becomes visible (after the occlusion) it is found again and the sampling function come back to be symmetric.

O rastreamento de objetos em vídeos representa um importante problema na área de processamento de imagens, quer seja pelo grande número de aplicações envolvidas, ou pelo grau de complexidade que pode ser apresentado. Como exemplo de aplicações, podemos citar sua utilização em áreas como robótica móvel, interface homem-máquina, medicina, automação de processo industriais até aplicações mais tracionais como vigilância e monitoramento de trafego. O aumento na complexidade do rastreamento se deve principalmente a interação do objeto rastreado com outros elementos da cena do vídeo, especialmente nos casos de oclusões parciais ou totais. Quando uma oclusão ocorre a informação sobre a localização do objeto durante o rastreamento é perdida parcial ou totalmente. Métodos de filtragem estocástica, utilizados para o rastreamento de objetos, como os Filtros de Partículas não apresentam resultados satisfatórios na presença de oclusões totais, onde temos uma descontinuidade na trajetória do objeto. Portanto torna-se necessário o desenvolvimento de métodos específicos para tratar o problema de oclusão total. Nesse trabalho, nós desenvolvemos uma abordagem para tratar o problema de oclusão total no rastreamento de objetos utilizando Filtro de Partículas baseados em Monte Carlo via Cadeia de Markov (MCCM) com função geradora de partículas adaptativa. Durante o rastreamento do objeto, em situações onde não há oclusões, nós utilizamos uma função de probabilidade geradora simétrica. Entretanto, quando uma oclusão total, ou seja, uma descontinuidade na trajetória é detectada, a função geradora torna-se assimétrica, criando um termo de “inércia” ou “arraste” na direção do deslocamento do objeto. Ao sair da oclusão, o objeto é novamente encontrado e a função geradora volta a ser simétrica novamente. / The object tracking on video is an important task in image processing area either for the great number of involved applications, or for the degree of complexity that can be presented. How example of application, we can cite its use from robotic area, machine-man interface, medicine, automation of industry process to vigilance and traffic control applications. The increase of complexity of tracking is occasioned principally by interaction of tracking object with other objects on video, specially when total or partial occlusions occurs. When a occlusion occur the information about the localization of tracking object is lost partially or totally. Stochastic filtering methods, like Particle Filter do not have satisfactory results in the presence of total occlusions. Total occlusion can be understood like discontinuity in the object trajectory. Therefore is necessary to develop specific method to handle the total occlusion task. In this work, we develop an approach to handle the total occlusion task using MCMC-Particle Filter with adaptive sampling probability function. When there is not occlusions we use a symmetric probability function to sample the particles. However, when there is a total occlusion, a discontinuity in the trajectory is detected, and the probability sampling function becomes asymmetric. This break of symmetry creates a “drift” or “inertial” term in object shift direction. When the tracking object becomes visible (after the occlusion) it is found again and the sampling function come back to be symmetric.

abstract: In UAVs and parking lots, it is typical to first collect an enormous number of pixels using conventional imagers. This is followed by employment of expensive methods to compress by throwing away redundant data. Subsequently, the compressed data is transmitted to a ground station. The past decade has seen the emergence of novel imagers called spatial-multiplexing cameras, which offer compression at the sensing level itself by providing an arbitrary linear measurements of the scene instead of pixel-based sampling. In this dissertation, I discuss various approaches for effective information extraction from spatial-multiplexing measurements and present the trade-offs between reliability of the performance and computational/storage load of the system. In the first part, I present a reconstruction-free approach to high-level inference in computer vision, wherein I consider the specific case of activity analysis, and show that using correlation filters, one can perform effective action recognition and localization directly from a class of spatial-multiplexing cameras, called compressive cameras, even at very low measurement rates of 1\%. In the second part, I outline a deep learning based non-iterative and real-time algorithm to reconstruct images from compressively sensed (CS) measurements, which can outperform the traditional iterative CS reconstruction algorithms in terms of reconstruction quality and time complexity, especially at low measurement rates. To overcome the limitations of compressive cameras, which are operated with random measurements and not particularly tuned to any task, in the third part of the dissertation, I propose a method to design spatial-multiplexing measurements, which are tuned to facilitate the easy extraction of features that are useful in computer vision tasks like object tracking. The work presented in the dissertation provides sufficient evidence to high-level inference in computer vision at extremely low measurement rates, and hence allows us to think about the possibility of revamping the current day computer systems. / Dissertation/Thesis / Doctoral Dissertation Electrical Engineering 2017

abstract: Motion capture using cost-effective sensing technology is challenging and the huge success of Microsoft Kinect has been attracting researchers to uncover the potential of using this technology into computer vision applications. In this thesis, an upper-body motion analysis in a home-based system for stroke rehabilitation using novel RGB-D camera - Kinect is presented. We address this problem by first conducting a systematic analysis of the usability of Kinect for motion analysis in stroke rehabilitation. Then a hybrid upper body tracking approach is proposed which combines off-the-shelf skeleton tracking with a novel depth-fused mean shift tracking method. We proposed several kinematic features reliably extracted from the proposed inexpensive and portable motion capture system and classifiers that correlate torso movement to clinical measures of unimpaired and impaired. Experiment results show that the proposed sensing and analysis works reliably on measuring torso movement quality and is promising for end-point tracking. The system is currently being deployed for large-scale evaluations. / Dissertation/Thesis / M.S. Electrical Engineering 2012

Les caméras de surveillance sont de plus en plus fréquemment présentes dans notre environnement (villes, supermarchés, aéroports, entrepôts, etc.). Ces caméras sont utilisées, entre autres, afin de pouvoir détecter des comportements suspects (intrusion par exemple) ou de reconnaître une catégorie d'objets ou de personnes (détection de genre, détection de plaques d'immatriculation par exemple). D'autres applications concernent également l'établissement de statistiques de fréquentation ou de passage (comptage d'entrée/sortie de personnes ou de véhicules) ou bien le suivi d'un ou plusieurs objets se déplaçant dans le champ de vision de la caméra (trajectoires d'objets, analyse du comportement des clients dans un magasin). Compte tenu du nombre croissant de caméras et de la difficulté à réaliser ces traitements manuellement, un ensemble de méthodes d'analyse vidéo ont été développées ces dernières années afin de pouvoir automatiser ces tâches. Dans cette thèse, nous nous concentrons essentiellement sur les tâches de détection et de suivi des objets mobiles à partir d'une caméra fixe. Contrairement aux méthodes basées uniquement sur les images acquises par les caméras, notre approche consiste à intégrer un certain nombre d'informations contextuelles à l'observation afin de pouvoir mieux interpréter ces images. Ainsi, nous proposons de construire un modèle géométrique et géolocalisé de la scène et de la caméra. Ce modèle est construit directement à partir des études de prédéploiement des caméras et peut notamment utiliser les données OpenStreetMap afin d'établir les modèles 3d des bâtiments proches de la caméra. Nous avons complété ce modèle en intégrant la possibilité de prédire la position du Soleil tout au long de la journée et ainsi pouvoir calculer les ombres projetées des objets de la scène. Cette prédiction des ombres a été mise à profit afin d'améliorer la segmentation des piétons par modèle de fond en supprimant les ombres du masque de mouvement. Concernant le suivi des objets mobiles, nous utilisons le formalisme des automates finis afin de modéliser efficacement les états et évolutions possibles d'un objet. Ceci nous permet d'adapter le traitement de chaque objet selon son état. Nous gérons les occultations inter-objets à l'aide d'un mécanisme de suivi collectif (suivi en groupe) des objets le temps de l'occultation et de ré-identification de ceux-ci à la fin de l'occultation. Notre algorithme s'adapte à n'importe quel type d'objet se déplaçant au sol (piétons, véhicules, etc.) et s'intègre naturellement au modèle de scène développé. Nous avons également développé un ensemble de "rétro-actions" tirant parti de la connaissance des objets suivis afin d'améliorer les détections obtenues à partir d'un modèle de fond. En particulier, nous avons abordé le cas des objets stationnaires, souvent intégrés à tort dans le fond, et avons revisité la méthode de suppression des ombres du masque de mouvement en tirant parti de la connaissance des objets suivis. L'ensemble des solutions proposées a été implémenté dans le logiciel de l'entreprise Foxstream et est compatible avec la contrainte d'exécution en temps réel nécessaire en vidéosurveillance. / Video-surveillance cameras are increasingly used in our environment. They are indeed present almost everywhere in the cities, supermarkets, airports, warehouses, etc. These cameras are used, among other things, in order to detect suspect behavior (an intrusion for instance) or to recognize a specific category of object or person (gender detection, license plates detection). Other applications also exist to count and/or track people in order to analyze their behavior. Due to the increasing number of cameras and the difficulty to achieve these tasks manually, several video analysis methods have been developed in order to address them automatically. In this thesis, we mainly focus on the detection and tracking of moving objects from a fixed camera. Unlike methods based solely on images captured by cameras, our approach integrates contextual pieces of information in order better interpret these images. Thus we propose to build a geometric and geolocalized model of the scene and the camera. This model is built directly from the pre-deployment studies of the cameras and uses the OpenStreetMap geographical database to build 3d models of buildings near the camera. We added to this model the ability to predict the position of the sun throughout the day and the resulting shadows in the scene. By predicting the shadows, and deleting them from the foreground mask, our method is able to improve the segmentation of pedestrians. Regarding the tracking of multiple mobile objects, we use the formalism of finite state machines to effectively model the states and possible transitions that an object is allowed to take. This allows us to tailor the processing of each object according to its state. We manage the inter-object occlusion using a collective tracking strategy. When taking part in an occlusion, objects are regrouped and tracked collectively. At the end of the occlusion, each object is re-identified and individual tracking resume. Our algorithm adapts to any type of ground-moving object (pedestrians, vehicles, etc.) and seamlessly integrates in the developed scene model. We have also developed several retro-actions taking advantage of the knowledge of tracked objects to improve the detections obtained with the background model. In particular, we tackle the issue of stationary objects often integrated erroneously in the background and we revisited the initial proposal regarding shadow removal. All proposed solutions have been implemented in the Foxstream products and are able to run in real-time.

Abstract Estimation of 2-D motion is one of the fundamental problems in video processing and computer vision. This thesis addresses two general tasks in estimating projected motions of background and foreground objects in a scene: global motion estimation and motion based segmentation. The work concentrates on the study of the block matching method, and especially on those cases where the matching measure is based on the sum of squared or absolute displaced frame differences. Related techniques for performing the confidence analysis of local displacement are considered and used to improve the performance of the higher-level tasks mentioned. In general, local motion estimation techniques suffer from the aperture problem. Therefore, confidence analysis methods are needed which can complement motion estimates with information about their reliability. This work studies a particular form of confidence analysis which uses the evaluation of the match criterion for local displacement candidates. In contrast to the existing approaches, the method takes into account the local image gradient. The second part of the thesis presents a four-step feature based method for global motion estimation. For basic observations, it uses motion features which are combinations of image point coordinates, displacement estimates at those points, and representations of displacement uncertainty. A parametric form of uncertainty representation is computed exploiting the technique described in the first part of the thesis. This confidence information is used as a basis for weighting the features in motion estimation. Aspects of gradient based feature point selection are also studied. In the experimental part, the design choices of the method are compared, using both synthetic and real sequences. In the third part of the thesis, a technique for feature based extraction of background and foreground motions is presented. The new sparse segmentation algorithm performs competitive segmentation using both the spatial and temporal propagation of support information. The weighting of features exploits parametric uncertainty information which is experimentally shown to improve the performance of motion estimation. In the final part of the thesis, a novel framework for motion based object detection, segmentation, and tracking is developed. It uses a block grid based representation for segmentation and a particle filter based approach to motion estimation. Analysis techniques for obtaining the segmentation are described. Finally, the approach is integrated with the sparse motion segmentation and the combination of the methods is experimentally shown to increase both the efficiency of sampling and the accuracy of segmentation. / Tiivistelmä Tässä väitöskirjassa tutkitaan yhtä videonkäsittelyn ja konenäön perusongelmaa, kaksiulotteisen liikkeen estimointia. Työ käsittelee kahta yleistä tehtävää taustan ja etualan kohteiden liikkeiden määrittämisessä: hallitsevan liikkeen estimointia ja liikepohjaista kuvan segmentointia. Tutkituissa ratkaisuissa lähtökohtana käytetään lohkosovitukseen perustuvaa paikallisen liikkeen määritystä, jossa sovituksen kriteerinä käytetään poikkeutettujen kehysten pikseliarvojen erotusta. Tähän liittyen tarkastellaan estimoinnin luotettavuuden analyysin tekniikoita ja näiden hyödyntämistä edellä mainittujen tehtävien ratkaisuissa. Yleensä ottaen paikallisen liikkeen estimointia vaikeuttaa apertuuriongelma. Tämän vuoksi tarvitaan analyysitekniikoita, jotka kykenevät antamaan täydentävää tietoa liike-estimaattien luotettavuudesta. Työn ensimmäisessä osassa kehitetty analyysimenetelmä käyttää lähtötietona lohkosovituksen kriteerin arvoja, jotka on saatu eri liikekandidaateille. Erotuksena aiempiin menetelmiin kehitetty ratkaisu ottaa huomioon kuvagradientin vaikutuksen. Työn toisessa osassa tutkitaan nelivaiheista piirrepohjaista ratkaisua hallitsevan liikkeen estimoimiseksi. Perushavaintoina mallissa käytetään liikepiirteitä, jotka koostuvat valittujen kuvapisteiden koordinaateista, näissä pisteissä lasketuista liike-estimaateista ja estimaattien epävarmuuden esityksestä. Jälkimmäinen esitetään parametrisessa muodossa käyttäen laskentaan työn ensimmäisessä osassa esitettyä menetelmää. Tätä epävarmuustietoa käytetään piirteiden painottamiseen hallitsevan liikkeen estimoinnissa. Lisäksi tutkitaan gradienttipohjaista piirteiden valintaa. Kokeellisessa osassa erilaisia suunnitteluvalintoja verrataan toisiinsa käyttäen synteettisiä ja todellisia kuvasekvenssejä. Väitöstyön kolmannessa osassa esitetään piirrepohjainen menetelmä taustan ja etualan kohteen liikkeiden erottamiseksi toisistaan. Algoritmi tekee analyysin kahta liikettä sisältävälle näkymälle käyttäen sekä spatiaalista että ajallista segmentointitiedon välittämistä. Piirteiden painotus hyödyntää epävarmuustietoa tässä yhteydessä, jonka osoitetaan kokeellisesti parantavan liike-estimoinnin suorituskykyä. Viimeisessä osassa kehitetään viitekehys liikepohjaisen kohteen ilmaisun, segmentoinnin ja seurannan toteutukselle. Se perustuu lohkopohjaiseen esitystapaan ja näytteistyksen soveltamiseen liikkeen estimoinnissa. Analyysitekniikka segmentoinnin määrittämiseksi esitellään. Lopuksi ratkaisu integroidaan työn kolmannessa osassa esitetyn menetelmän kanssa, ja menetelmien yhdistelmän osoitetaan kokeellisesti parantavan sekä näytteistyksen tehokkuutta että segmentoinnin tarkkuutta.

The computer vision problem of object tracking is introduced and explained. An approach to interest point based feature detection and tracking using FAST and BRIEF is presented and the selection of algorithms suitable for implementation on a Xilinx Zynq7000 with an XC7Z020 field-programmable gate array (FPGA) is detailed. A modification to the smoothing strategy of BRIEF which significantly reduces memory utilization on the FPGA is presented and benchmarked against a reference strategy. Measures of performance and resource efficiency are presented and utilized in an iterative development process. A system for interest point based object tracking that uses FAST for feature detection and BRIEF for feature description with the proposed smoothing modification is implemented on the FPGA. The design is described and important design choices are discussed.

A research field currently advancing is the use of machine learning on camera trap data, yet few explore deep learning for camera traps to be run in real-time. A camera trap has the purpose to capture images of bypassing animals and is traditionally based only on motion detection. This work integrates machine learning on the edge device to also perform object detection. Related research is brought up and model tests are performed with a focus on the trade-off regarding inference speed and model accuracy. Transfer learning is used to utilize pre-trained models and thus reduce training time and the amount of training data. Four models with slightly different architecture are compared to evaluate which model performs best for the use case. The models tested are SSD MobileNet V2, SSD Inception V2, and SSDLite MobileNet V2, SSD MobileNet V2 quantized. Since the client-side usage of the model, the SSD MobileNet V2 was finally selected due to a satisfying trade-off between inference speed and accuracy. Even though it is less accurate in its detections, its ability to detect more images per second makes it outperform the more accurate Inception network in object tracking. A contribution of this work is a light-weight tracking solution using tubelet proposal. This work further discusses the open set recognition problem, where just a few object classes are of interest while many others are present. The subject of open set recognition influences data collection and evaluation tests, it is however left for further work to research how to integrate support for open set recognition in object detection models. The proposed system handles detection, classification, and tracking of animals in the African savannah, and has potential for real usage as it produces meaningful events