The aim of this thesis is to predict targets in a Multiple Object Tracking (MOT) task, in which subjects track multiple moving objects. We processed and analyzed data containing object and gaze position information from 1148 MOT trials completed by 20 subjects. We extracted multiple features from the raw data and designed a machine learning approach for the prediction of targets using neural networks and hidden Markov models. We assessed the performance of the models and features. The results of our experiments show that it is possible to train a machine learning model to predict targets with very high accuracy. 1

The multiple object tracking task (Pylyshyn and Storm, 1988) has long been a standard tool for use in understanding how we attend to multiple moving points in the visual field. In the current experiments, it is first demonstrated that this classical task can be adapted for use in a simulated driving environment, where it is commonly thought to apply. Standard requirements of driving (steering, maintaining headway) are shown to reduce tracking ability. Subsequent experiments (2a, 2b, 2c) investigate the way in which participants respond to events at target and distractor locations, and have bearing on Pylyshyn’s (1989) “indexing” hypothesis. The final experiment investigates the effect of the colour-composition of the tracking set on performance, and may have implications for our theoretical understanding of how tracking is performed. / AUTO21, NSERC, CANDrive

Multiple object tracking consists of detecting and identifying objects in video. In some applications, such as robotics and surveillance, it is desired that the tracking is performed in real-time. This poses a challenge in that it requires the algorithm to run as fast as the frame-rate of the video. Today's top performing tracking methods run at only a few frames per second, and can thus not be used in real-time. Further, when determining the speed of the tracker, it is common to not include the time it takes to detect objects. We argue that this way of measuring speed is not relevant for robotics or embedded systems, where the detecting of objects is done on the same machine as the tracking. We propose that one way of running a method in real-time is to not look at every frame, but skip frames to make the video have the same frame-rate as the tracking method. However, we believe that this will lead to decreased performance. In this project, we implement a multiple object tracker, following the tracking-by-detection paradigm, as an extension of an existing method. It works by modelling the movement of objects by solving the filtering problem, and associating detections with predicted new locations in new frames using the Hungarian algorithm. Three different similarity measures are used, which use the location and shape of the bounding boxes. Compared to other trackers on the MOTChallenge leaderboard, our method, referred to as C++SORT, is the fastest non-anonymous submission, while also achieving decent score on other metrics. By running our model on the Okutama-Action dataset, sampled at different frame-rates, we show that the performance is greatly reduced when running the model - including detecting objects - in real-time. In most metrics, the score is reduced by 50%, but in certain cases as much as 90%. We argue that this indicates that other, slower methods could not be used for tracking in real-time, but that more research is required specifically on this. / För att spåra rörliga objekt i video (eng: multiple object tracking) krävs att man lokaliserar och identifierar dem. I vissa tillämpningar, såsom robotik och övervakning, kan det krävas att detta görs i realtid, vilket kan vara svårt i praktiken, då det förutsätter att algoritmen kan köras lika fort som videons bildfrekvensen. De kraftfullaste algoritmerna idag kan bara analysera ett fåtal bildrutor per sekund, och lämpar sig därför inte för realtidsanvändning. Dessutom brukar tiden per bildruta inte inkludera den tid det tar att lokalisera objekt, när hastigheten av en algoritm presenteras. Vi anser att det sättet att beräkna hastigheten inte är lämpligt inom robotik eller inbyggda system, där lokaliseringen och identifiering av objekt sker på samma maskin. Många algoritmer kan köras i realtid genom att hoppa över det antal bildrutor i videon som krävs för att bildfrekvensen ska bli densamma som algoritmens frekvens. Dock tror vi att detta leder till sämre prestanda. I det här projektet implementerar vi en algoritm för att identifiera rörliga objekt. Vår algoritm bygger på befintliga metoder inom paradigmen tracking-by-detection (ung. spårning genom detektion). Algoritmen uppskattar hastigheten hos varje objekt genom att lösa ett filtreringsproblem. Utifrån hastigheten beräknas en förväntad ny position, som kopplas till nya observationer med hjälp av Kuhn–Munkres algoritm. Tre olika likhetsmått används, som på olika sätt kombinerar positionen för och formen på objekten. Vår metod, C++SORT, är den snabbaste icke-anonyma metoden publicerad på MOTChallenge. Samtidigt presterar den bra enligt flera andra mått. Genom att testa vår algoritm på video från Okutama-Action, med varierande bildfrekvens, kan vi visa att prestandan sjunker kraftigt när hela modellen - inklusive att lokalisera objekt - körs i realtid. Prestandan enligt de flesta måtten sjunker med 50%, men i vissa fall med så mycket som 90%. Detta tyder på att andra, långsammare metoder inte kan användas i realtid, utan att mer forskning, specifikt inriktad på spårning i realtid, behövs.

No / Human observers can simultaneously track up to five targets in motion (Z. W. Pylyshyn & R. W. Storm, 1988). We examined the precision for detecting deviations in linear trajectories by measuring deviation thresholds as a function of the number of trajectories (T ). When all trajectories in the stimulus undergo the same deviation, thresholds are uninfluenced by T for T <= 10. When only one of the trajectories undergoes a deviation, thresholds rise steeply as T is increased [e.g., 3.3º (T = 1), 12.3º (T = 2), 32.9º (T = 4) for one observer]; observers are unable to simultaneously process more than one trajectory in our threshold-measuring paradigm. When the deviating trajectory is cued (e.g., using a different color), varying T has little influence on deviation threshold. The use of a different color for each trajectory does not facilitate deviation detection. Our current data suggest that for deviations that have low discriminability (i.e., close to threshold) the number of trajectories that can be monitored effectively is close to one. In contrast, when the stimuli containing highly discriminable (i.e., substantially suprathreshold) deviations are used, as many as three or four trajectories can be simultaneously monitored (S. P. Tripathy, 2003). Our results highlight a severe loss of positional information when attempting to track multiple objects, particularly in a threshold paradigm.

No / Amblyopia results in a severe loss of positional information and in the ability to accurately enumerate objects (V. Sharma, D. M. Levi, & S. A. Klein, 2000). In this study, we asked whether amblyopia also disrupts the ability to track a near-threshold change in the trajectory of a single target amongst multiple similar potential targets. In the first experiment, we examined the precision for detecting a deviation in the linear motion trajectory of a dot by measuring deviation thresholds as a function of the number of moving trajectories (T). As in normal observers, we found that in both eyes of amblyopes, threshold increases steeply as T increases from 1 to 4. Surprisingly, for T = 1-4, thresholds were essentially identical in both eyes of the amblyopes and were similar to those of normal observers. In a second experiment, we measured the precision for detecting a deviation in the orientation of a static, bilinear "trajectory" by again measuring deviation thresholds (i.e., angle discrimination) as a function of the number of oriented line "trajectories" (T). Relative to the nonamblyopic eye, amblyopes show a marked threshold elevation for a static target when T = 1. However, thresholds increased with T with approximately the same slope as in their preferred eye and in the eyes of the normal controls. We conclude that while amblyopia disrupts static angle discrimination, amblyopic dynamic deviation detection thresholds are normal or very nearly so.

The significant advancements and breakthroughs achieved in Machine Learning (ML) have revolutionized the field of Computer Vision (CV), where numerous real-world applications are now utilizing state-of-the-art advancements in the field. Advanced video surveillance and analytics, entertainment, and autonomous vehicles are a few examples that rely heavily on reliable and accurate perception systems. Deep learning usage in Computer Vision has come a long way since it sparked in 2012 with the introduction of Alexnet. Convolutional Neural Networks (CNN) have evolved to become more accurate and reliable. This is attributed to the advancements in GPU parallel processing, and to the recent availability of large scale and high quality annotated datasets that allow the training of complex models. However, ML models can only be as good as the data they train on and the data they receive in production. In real-world environments, a perception system often needs to be able to operate in different environments and conditions (weather, lighting, obstructions, etc.). As such, it is imperative for a perception system to utilize information from different types of sensors to mitigate the limitations of individual sensors. In this dissertation, we focus on studying the efficacy of using thermal sensors to enhance the robustness of perception systems. We focus on two common vision tasks: object detection and multiple object tracking. Through our work, we prove the viability of thermal sensors as a complement, and in some scenarios a replacement, to RGB cameras. For their important applications in autonomous vehicles and surveillance, we focus our research on pedestrian and vehicle perception. We also introduce the world's first (to the best of our knowledge) large scale dataset for pedestrian detection and tracking including thermal and corresponding RGB images.

No description available.

Attention allows us to monitor objects or regions of visual space and extract information from them to use for report or storage. Classical theories of attention assumed a single focus of selection but many everyday activities, such as playing video games, suggest otherwise. Nonetheless, the underlying mechanism which can explain the ability to divide attention has not been well established. Numerous attempts have been made in order to clarify divided attention, including analytical strategies as well as methods working with visual phenomena, even more sophisticated predictors incorporating information about past selection decisions. Virtually all the attempts approach this problem by constructing a simplified model of attention. In this study, we develop a version of the existing Bayesian framework to propose such models, and evaluate their ability to generate eye movement trajectories. For the comparison of models, we use the eye movement trajectories generated by several analytical strategies. We measure the similarity between...

Attention allows us to monitor objects or regions of visual space and extract information from them to use for report or storage. Classical theories of attention assumed a single focus of selection but many everyday activities, such as playing video games, suggest otherwise. Nonetheless, the underlying mechanism which can explain the ability to divide attention has not been well established. Numerous attempts have been made in order to clarify divided attention, including analytical strategies as well as methods working with visual phenomena, even more sophisticated predictors incorporating information about past selection decisions. Virtually all the attempts approach this problem by constructing a simplified model of attention. In this study, we develop a version of the existing Bayesian framework to propose such models, and evaluate their ability to generate eye movement trajectories. For the comparison of models, we use the eye movement trajectories generated by several analytical strategies. We measure the...

In everyday situations people have to track several objects at once (e.g. driving or collective sports). Multiple object tracking paradigm (MOT) plausibly simulate tracking several targets in laboratory conditions. When we track targets in tasks with many other objects in scene, it becomes difficult to discriminate objects in periphery (crowding). Although tracking could be done only using attention, it is interesting question how humans plan their eye movements during tracking. In our study, we conducted a MOT experiment in which we presented participants repeatedly several trials with varied number of distractors, we recorded eye movements and we measured consistency of eye movements using Normalized scanpath saliency (NSS) metric. We created several analytical strategies employing crowding avoidance and compared them with eye data. Beside analytical models, we trained neural networks to predict eye movements in MOT trial. The performance of the proposed models and neuron networks was evaluated in a new MOT experiment. The analytical models explained variability of eye movements well (results comparable to intraindividual noise in the data); predictions based on neural networks were less successful.