Multiple object tracking consists of detecting and identifying objects in video. In some applications, such as robotics and surveillance, it is desired that the tracking is performed in real-time. This poses a challenge in that it requires the algorithm to run as fast as the frame-rate of the video. Today's top performing tracking methods run at only a few frames per second, and can thus not be used in real-time. Further, when determining the speed of the tracker, it is common to not include the time it takes to detect objects. We argue that this way of measuring speed is not relevant for robotics or embedded systems, where the detecting of objects is done on the same machine as the tracking. We propose that one way of running a method in real-time is to not look at every frame, but skip frames to make the video have the same frame-rate as the tracking method. However, we believe that this will lead to decreased performance. In this project, we implement a multiple object tracker, following the tracking-by-detection paradigm, as an extension of an existing method. It works by modelling the movement of objects by solving the filtering problem, and associating detections with predicted new locations in new frames using the Hungarian algorithm. Three different similarity measures are used, which use the location and shape of the bounding boxes. Compared to other trackers on the MOTChallenge leaderboard, our method, referred to as C++SORT, is the fastest non-anonymous submission, while also achieving decent score on other metrics. By running our model on the Okutama-Action dataset, sampled at different frame-rates, we show that the performance is greatly reduced when running the model - including detecting objects - in real-time. In most metrics, the score is reduced by 50%, but in certain cases as much as 90%. We argue that this indicates that other, slower methods could not be used for tracking in real-time, but that more research is required specifically on this. / För att spåra rörliga objekt i video (eng: multiple object tracking) krävs att man lokaliserar och identifierar dem. I vissa tillämpningar, såsom robotik och övervakning, kan det krävas att detta görs i realtid, vilket kan vara svårt i praktiken, då det förutsätter att algoritmen kan köras lika fort som videons bildfrekvensen. De kraftfullaste algoritmerna idag kan bara analysera ett fåtal bildrutor per sekund, och lämpar sig därför inte för realtidsanvändning. Dessutom brukar tiden per bildruta inte inkludera den tid det tar att lokalisera objekt, när hastigheten av en algoritm presenteras. Vi anser att det sättet att beräkna hastigheten inte är lämpligt inom robotik eller inbyggda system, där lokaliseringen och identifiering av objekt sker på samma maskin. Många algoritmer kan köras i realtid genom att hoppa över det antal bildrutor i videon som krävs för att bildfrekvensen ska bli densamma som algoritmens frekvens. Dock tror vi att detta leder till sämre prestanda. I det här projektet implementerar vi en algoritm för att identifiera rörliga objekt. Vår algoritm bygger på befintliga metoder inom paradigmen tracking-by-detection (ung. spårning genom detektion). Algoritmen uppskattar hastigheten hos varje objekt genom att lösa ett filtreringsproblem. Utifrån hastigheten beräknas en förväntad ny position, som kopplas till nya observationer med hjälp av Kuhn–Munkres algoritm. Tre olika likhetsmått används, som på olika sätt kombinerar positionen för och formen på objekten. Vår metod, C++SORT, är den snabbaste icke-anonyma metoden publicerad på MOTChallenge. Samtidigt presterar den bra enligt flera andra mått. Genom att testa vår algoritm på video från Okutama-Action, med varierande bildfrekvens, kan vi visa att prestandan sjunker kraftigt när hela modellen - inklusive att lokalisera objekt - körs i realtid. Prestandan enligt de flesta måtten sjunker med 50%, men i vissa fall med så mycket som 90%. Detta tyder på att andra, långsammare metoder inte kan användas i realtid, utan att mer forskning, specifikt inriktad på spårning i realtid, behövs.

Multiple object tracking is the process of assigning unique and consistent identities to objects throughout a video sequence. A popular approach to multiple object tracking, and object tracking in general, is to use a method called tracking-by-detection. Tracking-by-detection is a two-stage procedure: an object detection algorithm first detects objects in a frame, these objects are then associated with already tracked objects by a tracking algorithm. One of the main concerns of this thesis is to investigate how different object detection algorithms perform on surveillance video supplied by National Forensic Centre. The thesis then goes on to explore how the stand-alone alone performance of the object detection algorithm correlates with overall performance of a tracking-by-detection system. Finally, the thesis investigates how the use of visual descriptors in the tracking stage of a tracking-by-detection system effects performance.  Results presented in this thesis suggest that the capacity of the object detection algorithm is highly indicative of the overall performance of the tracking-by-detection system. Further, this thesis also shows how the use of visual descriptors in the tracking stage can reduce the number of identity switches and thereby increase performance of the whole system.

Le suivi multi-objets, malgré les avancées récentes en détection d'objets, présente encore plusieurs difficultés spécifiques et reste ainsi une problématique difficile. Au cours de cette thèse nous proposons d'examiner l'emploi de représentations parcimonieuses au sein de méthodes de suivi multi-objets, dans le but d'améliorer les performances de ces dernières. La première contribution de cette thèse consiste à employer des représentations parcimonieuses collaboratives dans un système de suivi en ligne pour distinguer au mieux les cibles. Des représentations parcimonieuses structurées sont ensuite considérées pour s'adapter plus spécifiquement aux approches de suivi à fenêtre glissante. Une dernière contribution consiste à employer des dictionnaires denses, prenant en considération un grand nombre de positions non détectées au sein des images, de manière à être plus robuste vis-à-vis de la performance du détecteur d'objets employé. / Despite recent advances in object detection, multi-object tracking still raises some specific issues and therefore remains a challenging problem. In this thesis, we propose to investigate the use of sparse representations within multi-object tracking approaches in order to gain in performances. The first contribution of this thesis consists in designing an online tracking approach that takes advantage of collaborative sparse representations to better distinguish between the targets. Then, structured sparse representations are considered in order to be more suited to traking approaches based on a sliding window. In order to rely less on the object detector quality, we consider for the last contribution of this thesis to use dense dictionaries that are taking into account a large number of undetected locations inside each frame.

An autonomous vehicle is a complex system that requires a good perception of the surrounding environment to operate safely. One part of that is multiple object tracking, which is an essential component in camera-based perception whose responsibility is to estimate object motion from a sequence of images. This requires an association problem to be solved where newly estimated object positions are mapped to previously predicted trajectories, for which different solution strategies exist.  In this work, a multiple hypothesis tracking algorithm is implemented. The purpose is to demonstrate that measurement associations are improved compared to less compute-intensive alternatives. It was shown that the implemented algorithm performed 13 percent better than an intersection over union tracker when evaluated using a standard evaluation metric. Furthermore, this work also investigates the usage of abstraction layers to accelerate time-critical parallel operations on the GPU. It was found that the execution time of the tracking algorithm could be reduced by 42 percent by replacing four functions with implementations written in the purely functional array language Futhark. Finally, it was shown that a GPU code abstraction layer can reduce the knowledge barrier required to write efficient CUDA kernels.

Detecting generic objects is one important sensing task for applications that need to understand the environment, for example eXtended Reality (XR), drone navigation etc. However, Object Detection algorithms are particularly computationally heavy for real-time video analysis on resource-constrained mobile devices. Thus Object Tracking, which is a much lighter process, is introduced under the Tracking-By-Detection (TBD) paradigm to alleviate the computational overhead. Still, it is common that the configurations of the TBD remain unchanged, which would result in unnecessary computation and/or performance loss in many cases.\\ This Master's Thesis presents a novel approach for multi-objective optimization of the TBD process on precision and latency, with the platform being power-constrained devices. We propose a Deep Reinforcement Learning based scheduling architecture that selects appropriate TBD actions in video sequences to achieve the desired goals. Specifically, we develop a simulation environment providing Markovian state information as input for the scheduler neural network, justified options of TBD actions, and a scalarized reward function to combine the multiple objectives. Our results demonstrate that the trained policies can learn to utilize content information from the current and previous frames, thus optimally controlling the TBD process at each frame. The proposed approach outperforms the baselines that have fixed TBD configurations and recent research works, achieving the precision close to pure detection while keeping the latency much lower. Both tuneable configurations show positive and synergistic contribution to the optimization objectives. We also show that our policies are generalizable, with inference and action time of the scheduler having minimal latency overhead. This makes our scheduling design highly practical in real XR or similar applications on power-constrained devices. / Att upptäcka generiska objekt är en viktig uppgift inom avkänning för tillämpningar som behöver förstå omgivningen, såsom eXtended Reality (XR) och navigering med drönare, bland annat. Algoritmer för objektdetektering är dock särskilt beräkningstunga när det gäller videoanalyser i realtid på resursbegränsade mobila enheter. Objektspårning, å andra sidan, är en lättare process som vanligtvis implementeras under Tracking-By-Detection (TBD)-paradigmet för att minska beräkningskostnaden. Det är dock vanligt att TBD-konfigurationerna förblir oförändrade, vilket leder till onödig beräkning och/eller prestandaförlust i många fall.\\ I detta examensarbete presenteras en ny metod för multiobjektiv optimering av TBD-processen med avseende på precision och latens på plattformar med begränsad prestanda. Vi föreslår en djup förstärkningsinlärningsbaserad schemaläggningsarkitektur som väljer lämpliga TBD-åtgärder för videosekvenser för att uppnå de önskade målen. Vi utvecklar specifikt en simulering som tillhandahåller Markovian state-information som indata för schemaläggaren, samt neurala nätverk, motiverade alternativ för TBD-åtgärder och en skalariserad belöningsfunktion för att kombinera de olika målen. Våra resultat visar att de tränade strategierna kan lära sig att använda innehållsinformation från aktuella och tidigare ramar för att optimalt styra TBD-processen för varje bild. Det föreslagna tillvägagångssättet är bättre än både de grundläggande metoderna med en fast TBD-konfiguration och nyare forskningsarbeten. Det uppnår en precision som ligger nära den rena detektionen samtidigt som latensen hålls mycket låg. Båda justerbara konfigurationerna bidrar positivt och synergistiskt till optimeringsmålen. Vi visar också att våra strategier är generaliserbara genom att dela upp träning och testning med en 50 %-ig uppdelning, vilket resulterar i minimal inferenslatens och schemaläggarens handlingslatens. Detta gör vår schemaläggningsdesign mycket praktisk i verkliga XR- eller liknande tillämpningar på enheter med begränsad strömförsörjning.

Le travail développé dans cette thèse porte sur l'analyse de séquences vidéo. Cette dernière est basée sur 3 taches principales : la détection, la catégorisation et le suivi des objets. Le développement de solutions fiables pour l'analyse de séquences vidéo ouvre de nouveaux horizons pour plusieurs applications telles que les systèmes de transport intelligents, la vidéosurveillance et la robotique. Dans cette thèse, nous avons mis en avant plusieurs contributions pour traiter les problèmes de détection et de suivi d'objets multiples sur des séquences vidéo. Les techniques proposées sont basées sur l’apprentissage profonds et des approches de transfert d'apprentissage. Dans une première contribution, nous abordons le problème de la détection multi-objets en proposant une nouvelle technique de transfert d’apprentissage basé sur le formalisme et la théorie du filtre SMC (Sequential Monte Carlo) afin de spécialiser automatiquement un détecteur de réseau de neurones convolutionnel profond (DCNN) vers une scène cible. Dans une deuxième contribution, nous proposons une nouvelle approche de suivi multi-objets original basé sur des stratégies spatio-temporelles (entrelacement / entrelacement inverse) et un détecteur profond entrelacé, qui améliore les performances des algorithmes de suivi par détection et permet de suivre des objets dans des environnements complexes (occlusion, intersection, fort mouvement). Dans une troisième contribution, nous fournissons un système de surveillance du trafic, qui intègre une extension du technique SMC afin d’améliorer la précision de la détection de jour et de nuit et de spécialiser tout détecteur DCNN pour les caméras fixes et mobiles. Tout au long de ce rapport, nous fournissons des résultats quantitatifs et qualitatifs. Sur plusieurs aspects liés à l’analyse de séquences vidéo, ces travaux surpassent les cadres de détection et de suivi de pointe. En outre, nous avons implémenté avec succès nos infrastructures dans une plate-forme matérielle intégrée pour la surveillance et la sécurité du trafic routier. / The work developed in this PhD thesis is focused on video sequence analysis. Thelatter consists of object detection, categorization and tracking. The development ofreliable solutions for the analysis of video sequences opens new horizons for severalapplications such as intelligent transport systems, video surveillance and robotics.In this thesis, we put forward several contributions to deal with the problems ofdetecting and tracking multi-objects on video sequences. The proposed frameworksare based on deep learning networks and transfer learning approaches.In a first contribution, we tackle the problem of multi-object detection by puttingforward a new transfer learning framework based on the formalism and the theoryof a Sequential Monte Carlo (SMC) filter to automatically specialize a Deep ConvolutionalNeural Network (DCNN) detector towards a target scene. The suggestedspecialization framework is used in order to transfer the knowledge from the sourceand the target domain to the target scene and to estimate the unknown target distributionas a specialized dataset composed of samples from the target domain. Thesesamples are selected according to the importance of their weights which reflectsthe likelihood that they belong to the target distribution. The obtained specializeddataset allows training a specialized DCNN detector to a target scene withouthuman intervention.In a second contribution, we propose an original multi-object tracking frameworkbased on spatio-temporal strategies (interlacing/inverse interlacing) and aninterlaced deep detector, which improves the performances of tracking-by-detectionalgorithms and helps to track objects in complex videos (occlusion, intersection,strong motion).In a third contribution, we provide an embedded system for traffic surveillance,which integrates an extension of the SMC framework so as to improve the detectionaccuracy in both day and night conditions and to specialize any DCNN detector forboth mobile and stationary cameras.Throughout this report, we provide both quantitative and qualitative results.On several aspects related to video sequence analysis, this work outperformsthe state-of-the-art detection and tracking frameworks. In addition, we havesuccessfully implemented our frameworks in an embedded hardware platform forroad traffic safety and monitoring.

This master thesis deals with the analysis of the current possibilities for object tracking in the image, based on which is designed a procedure for creating a system capable of tracking an object of interest. Part of this work is designing virtual reality for the needs of implementation of the tracking system, which is finally deployed and tested on a real prototype of unmanned vehicle.

Cette dissertation traite du problème du suivi d'objets visuels, dont le but est de localiser un objet et de déterminer sa trajectoire au cours du temps. En particulier, nous nous concentrons sur les scénarios difficiles, dans lesquels les objets subissent d'importantes déformations et occlusions, ou quittent le champs de vision. A cette fin, nous proposons deux méthodes robustes qui apprennent un modèle pour l'objet d'intérêt et le mettent à jour, afin de refléter ses changements au cours du temps.Notre première méthode traite du problème du suivi dans le cas où les objets subissent d'importantes transformations géométriques comme une rotation ou un changement d'échelle. Nous présentons un nouvel algorithme de sélection de propositions, qui étend l'approche traditionnelle de ``suivi par détection''. Cette méthode procède en deux étapes: proposition puis sélection. Dans l'étape de proposition, nous construisons un ensemble de candidats qui représente les localisations potentielles de l'objet en estimant de manière robuste les transformations géométriques. La meilleure proposition est ensuite sélectionnée parmi cet ensemble de candidats pour précisément localiser l'objet en utilisant des indices d'apparence et de mouvement.Dans un second temps, nous traitons du problème de la mise à jour de modèles dans le suivi visuel, c'est-à-dire de déterminer quand il est besoin de mettre à jour le modèle de la cible, lequel peut subir une occlusion, ou quitter le champs de vision. Pour résoudre cela, nous utilisons des indices de mouvement pour identifier l'état d'un objet de manière automatique et nous mettons à jour le modèle uniquement lorsque l'objet est entièrement visible. En particulier, nous utilisons des trajectoires à long terme ainsi qu'une technique basée sur la coup de graphes pour estimer les parties de l'objet qui sont visibles.Nous avons évalué nos deux approches de manière étendue sur différents bancs d'essai de suivi, en particulier sur le récent banc d'essai de suivi en ligne et le jeu de donnée du concours de suivi visuel. Nos deux approches se comparent favorablement à l'état de l'art et font montre d'améliorations significatives par rapport à plusieurs autres récents suiveurs. Notre soumission au concours de suivi d'objets visuels de 2015 a par ailleurs remporté l'une de ces compétitions. / In this dissertation we address the problem of visual object tracking, whereinthe goal is to localize an object and determine its trajectory over time. Inparticular, we focus on challenging scenarios where the object undergoessignificant transformations, becomes occluded or leaves the field of view. Tothis end, we propose two robust methods which learn a model for the object ofinterest and update it, to reflect its changes over time.Our first method addresses the tracking problem in the context of objectsundergoing severe geometric transformations, such as rotation, change in scale.We present a novel proposal-selection algorithm, which extends the traditionaldiscriminative tracking-by-detection approach. This method proceeds in twostages -- proposal followed by selection. In the proposal stage, we compute acandidate pool that represents the potential locations of the object byrobustly estimating the geometric transformations. The best proposal is thenselected from this candidate set to localize the object precisely usingmultiple appearance and motion cues.Second, we consider the problem of model update in visual tracking, i.e.,determining when to update the model of the target, which may become occludedor leave the field of view. To address this, we use motion cues to identify thestate of the object in a principled way, and update the model only when theobject is fully visible. In particular, we utilize long-term trajectories incombination with a graph-cut based technique to estimate parts of the objectsthat are visible.We have evaluated both our approaches extensively on several trackingbenchmarks, notably, recent online tracking benchmark and the visual objecttracking challenge datasets. Both our approaches compare favorably to thestate of the art and show significant improvement over several other recenttrackers. Specifically, our submission to the visual object tracking challengeorganized in 2015 was the winner in one of the competitions.

Master's thesis addresses the long-term image tracking in video sequences. The project was intended to demonstrate the techniques that are needed for handling the long-term tracking. It primarily describes the techniques which application leads to construction of adaptive tracking system which is able to deal with the change of appearance of the object and unstable character of the surrounding environement appropriately.