Les interfaces cerveau-machine (ICMs) nous offrent un moyen de fermer la boucle entre notre cerveau et le monde de la technologie numérique. Cela ouvre la porte à une pléthore de nouvelles applications où nous utilisons directement le cerveau comme entrée. S’il est facile de voir le potentiel, il est moins facile de trouver la bonne application avec les bons corrélats neuronaux pour construire un tel système en boucle fermée. Ici, nous explorons une tâche de suivi d’objets multiples en 3D, dans un contexte d’entraînement cognitif (3D-MOT). Notre capacité à suivre plusieurs objets dans un environnement dynamique nous permet d’effectuer des tâches quotidiennes telles que conduire, pratiquer des sports d’équipe et marcher dans un centre commercial achalandé. Malgré plus de trois décennies de littérature sur les tâches MOT, les mécanismes neuronaux sous- jacents restent mal compris. Ici, nous avons examiné les corrélats neuronaux via l’électroencéphalographie (EEG) et leurs changements au cours des trois phases d’une tâche de 3D-MOT, à savoir l’identification, le suivi et le rappel. Nous avons observé ce qui semble être un transfert entre l’attention et la de mémoire de travail lors du passage entre le suivi et le rappel. Nos résultats ont révélé une forte inhibition des fréquences delta et thêta de la région frontale lors du suivi, suivie d’une forte (ré)activation de ces mêmes fréquences lors du rappel. Nos résultats ont également montré une activité de retard contralatérale (CDA en anglais), une activité négative soutenue dans l’hémisphère contralatérale aux positions des éléments visuels à suivre. Afin de déterminer si le CDA est un corrélat neuronal robuste pour les tâches de mémoire de travail visuelle, nous avons reproduit huit études liées au CDA avec un ensemble de données EEG accessible au public. Nous avons utilisé les données EEG brutes de ces huit études et les avons analysées avec le même pipeline de base pour extraire le CDA. Nous avons pu reproduire les résultats de chaque étude et montrer qu’avec un pipeline automatisé de base, nous pouvons extraire le CDA. Récemment, l’apprentissage profond (deep learning / DL en anglais) s’est révélé très prometteur pour aider à donner un sens aux signaux EEG en raison de sa capacité à apprendre de bonnes représentations à partir des données brutes. La question à savoir si l’apprentissage profond présente vraiment un avantage par rapport aux approches plus traditionnelles reste une question ouverte. Afin de répondre à cette question, nous avons examiné 154 articles appliquant le DL à l’EEG, publiés entre janvier 2010 et juillet 2018, et couvrant différents domaines d’application tels que l’épilepsie, le sommeil, les interfaces cerveau-machine et la surveillance cognitive et affective. Enfin, nous explorons la possibilité de fermer la boucle et de créer un ICM passif avec une tâche 3D-MOT. Nous classifions l’activité EEG pour prédire si une telle activité se produit pendant la phase de suivi ou de rappel de la tâche 3D-MOT. Nous avons également formé un classificateur pour les essais latéralisés afin de prédire si les cibles étaient présentées dans l’hémichamp gauche ou droit en utilisant l’activité EEG. Pour la classification de phase entre le suivi et le rappel, nous avons obtenu un 80% lors de l’entraînement d’un SVM sur plusieurs sujets en utilisant la puissance des bandes de fréquences thêta et delta des électrodes frontales. / Brain-computer interfaces (BCIs) offer us a way to close the loop between our brain and the digital world of technology. It opens the door for a plethora of new applications where we use the brain directly as an input. While it is easy to see the disruptive potential, it is less so easy to find the right application with the right neural correlates to build such closed-loop system. Here we explore closing the loop during a cognitive training 3D multiple object tracking task (3D-MOT). Our ability to track multiple objects in a dynamic environment enables us to perform everyday tasks such as driving, playing team sports, and walking in a crowded mall. Despite more than three decades of literature on MOT tasks, the underlying and intertwined neural mechanisms remain poorly understood. Here we looked at the electroencephalography (EEG) neural correlates and their changes across the three phases of a 3D-MOT task, namely identification, tracking and recall. We observed what seems to be a handoff between focused attention and working memory processes when going from tracking to recall. Our findings revealed a strong inhibition in delta and theta frequencies from the frontal region during tracking, followed by a strong (re)activation of these same frequencies during recall. Our results also showed contralateral delay activity (CDA), a sustained negativity over the hemisphere contralateral to the positions of visual items to be remembered. In order to investigate if the CDA is a robust neural correlate for visual working memory (VWM) tasks, we reproduced eight CDA-related studies with a publicly accessible EEG dataset. We used the raw EEG data from these eight studies and analysed all of them with the same basic pipeline to extract CDA. We were able to reproduce the results from all the studies and show that with a basic automated EEG pipeline we can extract a clear CDA signal. Recently, deep learning (DL) has shown great promise in helping make sense of EEG signals due to its capacity to learn good feature representations from raw data. Whether DL truly presents advantages as compared to more traditional EEG processing approaches, however, remains an open question. In order to address such question, we reviewed 154 papers that apply DL to EEG, published between January 2010 and July 2018, and spanning different application domains such as epilepsy, sleep, brain-computer interfacing, and cognitive and affective monitoring. Finally, we explore the potential for closing the loop and creating a passive BCI with a 3D-MOT task. We classify EEG activity to predict if such activity is happening during the tracking or the recall phase of the 3D-MOT task. We also trained a classifier for lateralized trials to predict if the targets were presented on the left or right hemifield using EEG brain activity. For the phase classification between tracking and recall, we obtained 80% accuracy when training a SVM across subjects using the theta and delta frequency band power from the frontal electrodes and 83% accuracy when training within subjects.

No description available.

Finding salient features in an image, and matching them to their corresponding features in another image is an important step for many vision-based applications. Feature description plays an important role in the feature matching process. A robust feature descriptor must works with a number of image deformations and should be computationally efficient. For resource-limited systems, floating point and complex operations such as multiplication and square root are not desirable. This research first introduces a robust and efficient feature descriptor called PRObability (PRO) descriptor that meets these requirements without sacrificing matching accuracy. The PRO descriptor is further improved by incorporating only affine features for matching. While performing well, PRO descriptor still requires larger descriptor size, higher offline computation time, and more memory space than other binary feature descriptors. SYnthetic BAsis (SYBA) descriptor is developed to overcome these drawbacks. SYBA is built on the basis of a new compressed sensing theory that uses synthetic basis functions to uniquely encode or reconstruct a signal. The SYBA descriptor is designed to provide accurate feature matching for real-time vision applications. To demonstrate its performance, we develop algorithms that utilize SYBA descriptor to localize the soccer ball in a broadcast soccer game video, track ground objects for unmanned aerial vehicle, and perform motion analysis, and improve visual odometry accuracy for advanced driver assistance systems. SYBA provides high feature matching accuracy with computational simplicity and requires minimal computational resources. It is a hardware-friendly feature description and matching algorithm suitable for embedded vision applications.

One of the things augmented reality depends on is object tracking, which is a problem classically found in cinematography and security. However, the algorithms designed for the classical application are often too expensive computationally or too complex to run on simpler mobile hardware. One of the methods to do object tracking is with a trained neural network, this has already led to great results but is unfortunately still running into some of the same problems as the classical algorithms. For this reason a neural network designed specifically for object tracking on mobile hardware needs to be developed. This thesis will propose two di erent neural networks designed for object tracking on mobile hardware. Both are based on a siamese network structure and methods to improve their accuracy using filtering are also introduced. The first network is a modified version of “CNN architecture for geometric matching” that utilizes an a ne regression to perform object tracking. This network was shown to underperform in the MOT benchmark as-well as the VOT benchmark and therefore not further developed. The second network is an object detector based on “SqueezeDet” in a siamese network structure utilizing the performance optimized layers of “MobileNets”. The accuracy of the object detector network is shown to be competitive in the VOT benchmark, placing at the 16th place compared to trackers from the 2016 challenge. It was also shown to run in real-time on mobile hardware. Thus the one shot object detection network used for a tracking application can improve the experience of augmented reality applications on mobile hardware.

Multiple object tracking (MOT) is the task of detecting multiple objects in a scene and associating detections over time to form tracks. It is essential for many scene understanding tasks like surveillance, robotics and autonomous driving. Nowadays, the dominating tracking pipeline is to first detect all individual objects in a scene followed by a separate data association step, also known as tracking-by-detection. Recently, methods doing simultaneous detection and tracking has emerged, combining the task of detection and tracking into one single framework. In this project, we analyse performance of multiple object tracking algorithms belonging to both tracking categories. The goal is to examine strengths, weaknesses, and real-time capability of different tracking approaches in order to understand their suitability in different applications. Results show that a tracking-by-detection system with Scaled-YOLOv4 and SORT achieves 46.8% accuracy at over 28 frames per second (FPS) on Nvidia GTX 1080. By reducing the input resolution, inference speed is increased to almost 50 FPS, making it well suitable for real-time application. The addition of a deep re-identification CNN reduces the number of identity switches by 47%. However, association speed drops as low as 14 FPS for densely populated scenes. This indicates that re-identification CNNs may be impractical for safety critical applications like autonomous driving, especially in urban environments. Simultaneous detection and tracking results suggests an increased tracking robustness. The removal of a complex data association strategy improves robustness with respect to extended modules like re-identification. This indicates that the inherent simplicity in the simultaneous detection and tracking paradigm can provide robust baseline trackers for a variety of applications. We note that further research is required to strengthen this notion. / Multipel objektspårning handlar om att detektera alla objekt i bilder och associera dem över tid för att bilda spår. Det är ett viktigt ämne inom datorseende för flera applikationer, däribland kameraövervakning, robotik och självkörande fordon. Idag är det dominerande tillvägagångsättet inom objektspårning att först detektera alla objekt och sedan associera dem i ett separat steg, också kallat spårning-genom-detektion. På senare tid har det framkommit nya metoder som detekterar och spårar samtidigt. I detta projekt analyserar vi prestanda av metoder som tillämpar båda tillvägagångssätt. Målet med projektet är att undersöka styrkor, svagheter och hur väl metoderna lämpar sig för att användas i realtid. Detta för att förstå hur olika objektspårare kan anpassas till olika praktiska applikationer. Resultaten visar att ett system som tillämpar spårning-genom-detektion med Scaled-YOLOv4 och SORT, uppnår 46.8% noggrannhet med en hastighet på över 28 bildrutor per sekund. Detta på en Nvidia GTX 1080. Genom att minska bildupplösningen når hastigheten nästan hela vägen upp till 50 bildrutor per sekund, vilket gör systemet väl lämpat för realtidsapplikation. Genom att addera ett djupt nätverk för återidentifiering minskar antalet identitetsbyten med 47%. Samtidigt minskar också hastigheten för spårning till 14 bildrutor per sekund i välbefolkade miljöer. Detta indikerar att djupa nätverk för återidentifiering inte lämpar sig för säkerhetskritiska applikationer såsom självkörande fordon. Särskilt i urbana miljöer. Resultat för system som detekterar och spårar samtidigt antyder att de är mer robusta. Genom att ta bort komplexa strategier för associering blir systemen robusta mot ytterligare moduler såsom återidentifiering. Det ger en indikation på att den inneboende enkelheten i dessa system resulterar i objektspårare som kan fungera som grunder i många olika applikationer. Vi noterar att ytterligare forsking behövs för att styrka denna idé.

This thesis focuses on visual multiple-target tracking (MTT) from a UAV. Typical state-of-the-art multiple-target trackers rely on an object detector as the primary detection source. However, object detectors usually require a GPU to process images in real-time, which may not be feasible to carry on-board a UAV. Additionally, they often do not produce consistent detections for small objects typical of UAV imagery.In our method, we instead detect motion to identify objects of interest in the scene. We detect motion at corners in the image using optical flow. We also track points long-term to continue tracking stopped objects. Since our motion detection algorithm generates multiple detections at each time-step, we use a hybrid probabilistic data association filter combined with a single iteration of expectation maximization to improve tracking accuracy.We also present a motion detection algorithm that accounts for parallax in non-planar UAV imagery. We use the essential matrix to distinguish between true object motion and apparent object motion due to parallax. Instead of calculating the essential matrix directly, which can be time-consuming, we design a new algorithm that optimizes the rotation and translation between frames. This new algorithm requires only 4 ms instead of 47 ms per frame of the video sequence.We demonstrate the performance of these algorithms on video data. These algorithms are shown to improve tracking accuracy, reliability, and speed. All these contributions are capable of running in real-time without a GPU.

Indiana University-Purdue University Indianapolis (IUPUI) / Simultaneous localization and mapping (SLAM) is a general device localization technique that uses realtime sensor measurements to develop a virtualization of the sensor's environment while also using this growing virtualization to determine the position and orientation of the sensor. This is useful for augmented reality (AR), in which a user looks through a head-mounted display (HMD) or viewfinder to see virtual components integrated into the real world. Visual SLAM (i.e., SLAM in which the sensor is an optical camera) is used in AR to determine the exact device/headset movement so that the virtual components can be accurately redrawn to the screen, matching the perceived motion of the world around the user as the user moves the device/headset. However, many potential AR applications may need access to more than device localization data in order to be useful; they may need to leverage environment data as well. Additionally, most SLAM solutions make the naive assumption that the environment surrounding the system is completely static (non-moving). Given these circumstances, it is clear that AR may benefit substantially from utilizing a SLAM solution that detects objects that move in the scene and ultimately provides localization data for each of these objects. This problem is known as the dynamic SLAM problem. Current attempts to address the dynamic SLAM problem often use machine learning to develop models that identify the parts of the camera image that belong to one of many classes of potentially-moving objects. The limitation with these approaches is that it is impractical to train models to identify every possible object that moves; additionally, some potentially-moving objects may be static in the scene, which these approaches often do not account for. Some other attempts to address the dynamic SLAM problem also localize the moving objects they detect, but these systems almost always rely on depth sensors or stereo camera configurations, which have significant limitations in real-world use cases. This dissertation presents a novel approach for registering and localizing unknown moving objects in the context of markerless, monocular, keyframe-based SLAM with no required prior information about object structure, appearance, or existence. This work also details a novel deep learning solution for determining SLAM map initialization suitability in structure-from-motion-based initialization approaches. This dissertation goes on to validate these approaches by implementing them in a markerless, monocular SLAM system called LUMO-SLAM, which is built from the ground up to demonstrate this approach to unknown moving object registration and localization. Results are collected for the LUMO-SLAM system, which address the accuracy of its camera localization estimates, the accuracy of its moving object localization estimates, and the consistency with which it registers moving objects in the scene. These results show that this solution to the dynamic SLAM problem, though it does not act as a practical solution for all use cases, has an ability to accurately register and localize unknown moving objects in such a way that makes it useful for some applications of AR without thwarting the system's ability to also perform accurate camera localization.

The integration of robotic modules is common both in industry and in academia, especially when it comes to robotic grasping and object tracking. However, there are usually two challenges in the integration process. Firstly, the respective fields are extensive, making it challenging to select a method in each field for integration according to specific needs. Secondly, because the integrated system is rarely discussed in academia, there is no set of metrics to evaluate it. Aiming at the first challenge, this thesis reviews and categorizes popular methods in the fields of robotic grasping and object tracking, summarizing their advantages and disadvantages. This categorization provides the basis for selecting methods according to the specific needs of application scenarios. For evaluation, two well-established methods for grasp pose detection and object tracking are integrated for a common application scenario. Furthermore, the technical, as well as the task-related challenges of the integration process are discussed. Finally, in response to the second challenge, a set of metrics is proposed to evaluate the integrated system. / Integration av robotmoduler är vanligt förekommande både inom industrin och i den akademiska världen, särskilt när det gäller robotgrepp och objektspårning. Det finns dock vanligtvis två utmaningar i integrationsprocessen. För det första är både respektive fält omfattande, vilket gör det svårt att välja en metod inom varje fält för integration enligt relevanta behov. För det andra, eftersom det integrerade systemet sällan diskuteras i den akademiska världen, finns det inga etablerade mätvärden för att utvärdera det. För att fokusera på den första utmaningen, granskar och kategoriserar denna avhandling populära metoder inom robotgreppning och spårning av objekt, samt sammanfattar deras fördelar och nackdelar. Denna kategorisering utgör grunden för att välja metoder enligt de specifika behoven i olika applikationsscenarion. Som utvärdering, integreras samt jämförs två väletablerade metoder för posdetektion och objektspårning för ett vanligt applikationsscenario. Vidare diskuteras de tekniska och uppgiftsrelaterade utmaningarna i integrationsprocessen. Slutligen, som svar på den andra utmaningen, föreslås en uppsättning mätvärden för att utvärdera det integrerade systemet.

Location-based services (LBSs) have become pervasive, and the demand for these systems and services is rising. Indoor Positioning Systems (IPSs) are key to extend location-based services indoors where the Global Positioning System (GPS) is not reliable due to low signal strength and complicated signal propagation environment. Most existing IPSs either require the installation of special hardware devices or build a fingerprint map, which is expensive, time-consuming, and labor-intensive. Developments in microelectromechanical systems (MEMS) have resulted in significant advancements in the low-cost compact MARG inertial sensors, making it possible to achieve low-cost and high-accuracy IPSs. This research considers the indoor positioning problem and aims to design and develop an infrastructure-free self-contained indoor positioning and tracking system based on Pedestrian Dead Reckoning (PDR) using MEMS MARG inertial sensors. PDR-based systems rely on MARG inertial sensor measurements to estimate the current position of the object by using a previously determined position without external references. Many issues still exist in developing such systems, such as cumulative errors, high-frequency sensor noises, the gyro drift issue, magnetic distortions, etc. As the MARG sensors are inherently error-prone, the most significant challenge is how to design sensor fusion models and algorithms to accurately extract useful location-based information from individual motion and magnetic sensors. The objective of this thesis is to solve these issues and mitigate the challenges. The proposed positioning system is designed with four main modules at the system level and a dual-mode feature. Specifically, the four main modules are mode detection, step detection and moving distance estimation, heading and orientation estimation, and position estimation. To address the cumulative error issue of using low-cost inertial sensors, signal processing and sensor fusion techniques are utilized for algorithm design. Experimental evaluations show that the proposed position estimation algorithm is able to achieve high positioning accuracy at low costs for the indoor environment. / Thesis / Master of Applied Science (MASc) / With the maturity of microelectromechanical systems (MEMS) technology in recent years, Magnetic, Angular Rate, and Gravity (MARG) sensors are embedded in most smart devices. This research considers the indoor positioning problem and aims to design and develop an infrastructure-free self-contained MEMS MARG inertial sensor-based indoor positioning and tracking system with high precision. The proposed positioning system uses the Pedestrian Dead Reckoning (PDR) approach and includes four main modules at the system level with a dual-mode feature. Specifically, the four main modules are mode detection, step detection and moving distance estimation, heading and orientation estimation, and position estimation. The two modes are static mode and dynamic mode. To address the cumulative error issue of using low-cost inertial sensors, signal processing and sensor fusion techniques are utilized for algorithm design. The detection and estimation algorithms of each module are presented in the system design chapter. Experimental evaluations including trajectory results under five scenarios show that the proposed position estimation algorithm achieves a higher position accuracy than that of conventional estimation methods.

No description available.