As climate change concerns grow, the reduction of energy consumption is seen as one of many potential solutions. In the US, a considerable amount of energy is wasted in commercial buildings due to sub-optimal heating, ventilation and air conditioning that operate with no knowledge of the occupancy level in various rooms and open areas. In this thesis, I develop an approach to passive occupancy estimation that does not require occupants to carry any type of beacon, but instead uses an overhead camera with fisheye lens (360 by 180 degree field of view). The difficulty with fisheye images is that occupants may appear not only in the upright position, but also upside-down, horizontally and diagonally, and thus algorithms developed for typical side-mounted, standard-lens cameras tend to fail. As the top-performing people detection algorithms today use deep learning, a logical step would be to develop and train a new neural-network model. However, there exist no large fisheye-image datasets with person annotations to facilitate training a new model. Therefore, I developed two people-counting methods that leverage YOLO (version 3), a state-of-the-art object detection method trained on standard datasets. In one approach, YOLO is applied to 24 rotated and highly-overlapping windows, and the results are post-processed to produce a people count. In the other approach, regions of interest are first extracted via background subtraction and only windows that include such regions are supplied to YOLO and post-processed. I carried out extensive experimental evaluation of both algorithms and showed their superior performance compared to a benchmark method.

Η διάδοση της χρήσης των υπολογιστών σε όλο και περισσότερους τομείς της καθημερινής μας ζωής, καθώς και η τεχνολογική εξέλιξη στην επιστήμη των υπολογιστών είχε σαν φυσικό επακόλουθο τη δημιουργία αλγορίθμων που έχουν στόχο την ανίχνευση και την αναγνώριση ανθρώπων με ακρίβεια καθώς και την παρακολούθηση τους. Τέτοιοι αλγόριθμοι εφαρμόζονται κυρίως σε συστήματα οπτικής επιτήρησης που είναι ζωτικής σημασίας σε διάφορους τομείς της καθημερινότητας. Αντικείμενο της παρούσας διπλωματικής εργασίας είναι η υλοποίηση ενός συστήματος ανίχνευσης, με τη χρήση του αλγόριθμου Histogram of Oriented Gradient (HOG), ταξινόμησης με χρήση Supported Vector Machines και παρακολούθησης ανθρώπου σε ακολουθία εικόνων, με χρήση αλγορίθμων υπολογιστικής όρασης όπως είναι ο αλγόριθμος φιλτραρίσματος σωματιδίων (Particle Filtering). / The widespread use of computers in more and more areas of our everyday life and the technological development in computer science as a natural consequence was the creation of algorithms that aim to detect and identify people accurately and monitor them. Such algorithms, are applied mainly in visual surveillance systems and is of vital importance in various areas of everyday life. The subject of this thesis is to implement a detection system using the algorithm Histogram of Oriented Gradient (HOG) as well, sort using Supported Vector Machines and the human tracking in image sequence, using computer vision algorithms such as Particle Filtering algorithm.

People detection has been an attractive technology in computer vision. There are many useful applications in our daily life, for instance, intelligent surveillance and driver assistance system. People detection is a challenging matter as people adopt a wide range of poses, wear diverse clothes, and are visible in different kind of backgrounds with significant changes in illumination. In this thesis, some advanced techniques and powerful tools are presented in order to design a robust people detection system. First a baseline model is implemented by combining the Histogram of Oriented Gradients descriptor and linear Support Vector Machines. This baseline model obtains a good performance on the well-known INRIA dataset. Second an advanced model is proposed which has a two-layer cascade framework that achieves both accurate detection and lower computational complexity. For the first layer, the baseline model is used as a filter to generate several candidates. In this procedure, most positive samples survived and the majority of negative samples are rejected according to a preset threshold. The second layer uses a more discriminative model. We combine the Variational Local Binary Patterns descriptor, and the Histogram of Oriented Gradients descriptor as a new discriminative feature. Furthermore multi-scale feature descriptors are used to improve the discriminative power of the Variational Local Binary Patterns feature. Then we perform Feature Selection using the Feature Generating Machine in order to generate a concise descriptor based on this concatenated feature. Moreover Histogram Intersection Kernel Support Vector Machines is employed as an efficient tool of classification. The bootstrapping algorithm is used in the training procedure to exploit the information of the dataset. Finally our approach has a good performance on the INRIA dataset, with results superior to the baseline model.

京都大学 / 0048 / 新制・課程博士 / 博士(情報学) / 甲第17926号 / 情博第508号 / 新制||情||90(附属図書館) / 30746 / 京都大学大学院情報学研究科知能情報学専攻 / (主査)教授 奥乃 博, 教授 河原 達也, 教授 中村 裕一, 教授 五十嵐 淳 / 学位規則第4条第1項該当 / Doctor of Informatics / Kyoto University / DFAM

This thesis presents a general, trainable system for object detection in static images and video sequences. The core system finds a certain class of objects in static images of completely unconstrained, cluttered scenes without using motion, tracking, or handcrafted models and without making any assumptions on the scene structure or the number of objects in the scene. The system uses a set of training data of positive and negative example images as input, transforms the pixel images to a Haar wavelet representation, and uses a support vector machine classifier to learn the difference between in-class and out-of-class patterns. To detect objects in out-of-sample images, we do a brute force search over all the subwindows in the image. This system is applied to face, people, and car detection with excellent results. For our extensions to video sequences, we augment the core static detection system in several ways -- 1) extending the representation to five frames, 2) implementing an approximation to a Kalman filter, and 3) modeling detections in an image as a density and propagating this density through time according to measured features. In addition, we present a real-time version of the system that is currently running in a DaimlerChrysler experimental vehicle. As part of this thesis, we also present a system that, instead of detecting full patterns, uses a component-based approach. We find it to be more robust to occlusions, rotations in depth, and severe lighting conditions for people detection than the full body version. We also experiment with various other representations including pixels and principal components and show results that quantify how the number of features, color, and gray-level affect performance.

Les travaux présentés dans ce manuscrit traitent de la détection de personnes dans des séquences d’images et de l’analyse de leur activité. Ces travaux ont été menés au sein de l’institut PRISME dans le cadre du projet CAPTHOM du pôle de compétitivité S2E2. Après un état de l’art sur l’analyse de séquences d’images pour l’interprétation automatique de scènes et une étude comparative de modules de vidéo-surveillance, nous présentons la méthode de détection de personnes proposée dans le cadre du projet CAPTHOM. Celle-ci s’articule autour de trois étapes : la détection de changement, le suivi d’objets mobiles et la classification. Chacune de ces étapes est décrite dans ce manuscrit. Ce système a été évalué sur une large base de vidéos correspondant à des scénarios de cas d’usage de CAPTHOM établis par les partenaires du projet. Ensuite, nous présentons des méthodes permettant d’obtenir, à partir du flux vidéo d’une ou deux caméras, d’autres informations de plus haut-niveau sur l’activité des personnes détectées. Nous présentons tout d’abord une mesure permettant de quantifier leur activité. Ensuite, un système de stéréovision multi-capteurs combinant une caméra infrarouge et une caméra visible est utilisé pour augmenter les performances du système de détection mais aussi pour permettre la localisation dans l’espace des personnes et donc accéder à une cartographie de leurs déplacements. Finalement, une méthode de détection d’événements anormaux, basée sur des statistiques de distributions spatiales et temporelles des pixels de l’avant-plan est détaillée. Les méthodes proposées offrent un panel de solutions performantes sur l’extraction d’informations haut-niveau à partir de séquences d’images. / The work presented in this manuscript deals with people detection and activity analysis in images sequences. This work has been done in the PRISME institut within the framework of the CAPTHOM project of the French Cluster S2E2. After a state of the art on video analysis and a comparative study of several video surveillance tools, we present the people detection method proposed within the framework of the CAPTHOM project. This method is based on three steps : change detection, mobile objects tracking and classification. Each steps is described in this thesis. The system was assessed on a wide videos dataset. Then, we present methods used to obtain other high-level information concerning the activity of detected persons. A criterion for characterizing their activity is presented. Then, a multi-sensors stereovision system combining an infrared and a daylight camera is used to increase performances of the people detection system but also to localize persons in the 3D space and so build the moving cartography. Finally, an abnormal events detection method based on statistics about spatio-temporal foreground pixel distribution is presented. These proposed methods offer robust and efficient solutions on high-level information extraction from images sequences.

Cette thèse traite de la détection et de la ré-identification de personnes dans un environnement instrumenté par un réseau de caméras à champ disjoint. Elle est à la confluence des communautés Recherche Opérationnelle et Vision car elle s'appuie sur des techniques d'optimisation combinatoire pour formaliser de nouvelles modalités de vision par ordinateur. Dans ce contexte, un détecteur visuel de personnes, basé sur la programmation linéaire en nombres entiers, est tout d'abord proposé. Son originalité est de prendre en compte le coût de traitement et non uniquement les performances de détection. Ce détecteur est évalué et comparé aux détecteurs de la littérature les plus performants. Ces expérimentations menées sur deux bases de données publiques mettent clairement en évidence l'intérêt de notre détecteur en terme de coût de traitement avec garantie de performance de détection. La seconde partie de la thèse porte sur la modalité de ré-identification de personnes. L'originalité de notre approche, dénommée D-NCR (pour Directed Network Consistent Re-identification), est de prendre explicitement en compte les temps minimum de transit des personnes dans le réseau de caméras et sa topologie pour améliorer la performance de la ré-identification. On montre que ce problème s'apparente à une recherche de chemins disjoints particuliers à profit maximum dans un graphe orienté. Un programme linéaire en nombres entiers est proposé pour sa modélisation et résolution. Les évaluations réalisées sur une base publique d'images sont prometteuses et montrent le potentiel de cette approche. / This thesis deals with people detection and re-identification in an environment instrumented by a network of disjoint-field cameras. It stands at the confluence of the Operational Research and Computer Vision communities as combinatorial optimization techniques are used to formalize new computer vision methods. In this context, a people visual detector, based on mixed-integer programming, is first propose that simultaneously take computation time and detection performances into account. This detector is evaluated and compared to the best detectors of the literature. These experiments, conducted on two public databases, clearly demonstrate the interest of our detector in terms of processing time with detection performance guarantee. The second part of the thesis deals with people re-identification. Our novel approach, called D-NCR (Directed Network Consistent Re-identification), explicitly takes minimum transit times in the camera network into account, as well as the network topology, in order to improve the re-identification performance. This problem is similar to the determination of particular maximum-profitable independent paths in an oriented graph. A mixed-integer program is proposed to model and solve this problem. The experiments made on a public dataset sound promising and tend to prove the potential of the approach.

Thesis (MSc)--Stellenbosch University, 2014. / ENGLISH ABSTRACT: In this thesiswe approach the problem of tracking multiple people individually in a video sequence. Automatic object detection and tracking is non-trivial as humans have complex and mostly unpredictable movements, and there are sensor noise and measurement uncertainties present. We consider traditional object detection methods and decide to use thermal data for the detection step. This choice is supported by the robustness of thermal data compared to colour data in unfavourable lighting conditions and in surveillance applications. A drawback of using thermal data is that we lose colour information, since the sensor interprets the heat emission of the body rather than visible light. We incorporate a colour sensor which is used to build features for each detected object. These features are used to help determine correspondences in detected objects over time. A problem with traditional blob detection algorithms, which typically consist of background subtraction followed by connected-component labelling, is that objects can appear to split or merge, or disappear in a few frames. We decide to add ‘dummy’ blobs in an effort to counteract these problems. We refrain from making any hard decisions with respect to the blob correspondences over time, and rather let the system decide which correspondences are more probable. Furthermore, we find that the traditional Markovian approach of determining correspondences between detected blobs in the current time step and only the previous time step can lead to unwanted behaviour. We rather consider a sequence of time steps and optimize the tracking across them. We build a composite correspondence model and weigh each correspondence according to similarity (correlation) in object features. All possible tracks are determined through this model and a likelihood is calculated for each. Using the best scoring tracks we then label all the detections and use this labelling as measurement input for a tracking filter. We find that the window tracking approach shows promise even though the data we us for testing is of poor quality and noisy. The system struggles with cluttered scenes and when a lot of dummy nodes are present. Nonetheless our findings act as a proof of concept and we discuss a few future improvements that can be considered. / AFRIKAANSE OPSOMMING: In hierdie tesis benader ons die probleemomverskeiemense individueel in ’n video-opname op te spoor en te volg. Outomatiese voorwerp-opsporing en -volging is nie-triviaal, want mense het komplekse en meestal onvoorspelbare bewegings, en daar is sensor-ruis en metingonsekerhede teenwoordig. Ons neem tradisionele voorwerp-opsporing metodes in ag en besluit om termiese data te gebruik vir die opsporingstap. Hierdie keuse word ondersteun deur die robuustheid van termiese data in vergelyking met kleur data in ongunstige lig-kondisies en in sekuriteitstoepassings. Die nadeel van die gebruik van termiese data is dat ons kleur inligting verloor, aangesien die sensor die hitte vrystelling van die liggaam interpreteer, eerder as sigbare lig. Ons inkorporeer ’n kleur-sensor wat gebruik word om die kenmerke van elke gevolgde voorwerp te bou. Hierdie kenmerke word gebruik om te help om ooreenkomste tussen opgespoorde voorwerpe te bepaal met die verloop van tyd. ’n Probleem met die tradisionele voorwerp-opsporing algoritmes, wat tipies bestaan uit agtergrond- aftrekking gevolg deur komponent-etikettering, is dat dit kan voorkom asof voorwerpe verdeel of saamsmelt, of verdwyn in ’n paar rame. Ons besluit om ‘flous’-voorwerpe by te voeg in ’n poging om hierdie probleme teen te werk. Ons weerhou om enige konkrete besluite oor opgespoorde voorwerpe se ooreenkomste met die verloop van tyd te maak, en laat die stelsel eerder toe om te besluit watter ooreenkomste meer waarskynlik is. Verder vind ons dat die tradisionele Markoviaanse benadering vir die bepaling van ooreenkomste tussen opgespoorde voorwerpe in die huidige tydstap en die vorige een kan lei tot ongewenste gedrag. Ons oorweeg eerder ’n reeks van tydstappe, of ’n venster, en optimeer die volg van voorwerpe oor hulle. Ons bou ’n saamgestelde ooreenstemmingsmodel en weeg elke ooreenstemming volgens die ooreenkoms (korrelasie) tussen voorwerpe se kenmerke. Alle moontlike spore word deur hierdie model bepaal en ’n waarskynlikheid word bereken vir elkeen. Die spore met die beste tellings word gebruik om al die opsporings te nommeer, en hierdie etikettering word gebruik as meting-inset vir ’n volgingsfilter. Ons vind dat die venster-volg benadering belowend vaar selfs al is die invoerdata in ons toetse van swak gehalte en ruiserig. Die stelsel sukkel met besige tonele en wanneer baie flous-voorwerpe teenwoordig is. Tog dien ons bevindinge as ’n bewys van konsep en ons bespreek ’n paar verbeterings wat in die toekoms oorweeg kan word.

Os robôs móveis e de serviço vêm assumindo um papel cada vez mais amplo e importante junto à sociedade moderna. Um tipo importante de robô móvel autônomo são os robôs voltados para a vigilância e segurança em ambientes internos (indoor). Estes robôs móveis de vigilância permitem a execução de tarefas repetitivas de monitoramento de ambientes, as quais podem inclusive apresentar riscos à integridade física das pessoas, podendo assim ser executadas de modo autônomo e seguro pelo robô. Este trabalho teve por objetivo o desenvolvimento dos principais módulos que compõem a arquitetura de um sistema robótico de vigilância, que incluem notadamente: (i) a aplicação de sensores com percepção 3D (Kinect) e térmica (Câmera FLIR), de relativo baixo custo, junto a este sistema robótico; (ii) a detecção de intrusos (pessoas) através do uso conjunto dos sensores 3D e térmico; (iii) a navegação de robôs móveis autônomos com detecção e desvio de obstáculos, para a execução de tarefas de monitoramento e vigilância de ambientes internos; (iv) a identificação e reconhecimento de elementos do ambiente que permitem ao robô realizar uma navegação baseada em mapas topológicos. Foram utilizados métodos de visão computacional, processamento de imagens e inteligência computacional para a realização das tarefas de vigilância. O sensor de distância Kinect foi utilizado na percepção do sistema robótico, permitindo a navegação, desvio de obstáculos, e a identificação da posição do robô em relação a um mapa topológico utilizado. Para a tarefa de detecção de pessoas no ambiente foram utilizados os sensores Kinect e câmera térmica FLIR, integrando os dados fornecidos por ambos sensores, e assim, permitindo obter uma melhor percepção do ambiente e também permitindo uma maior confiabilidade na detecção de pessoas. Como principal resultado deste trabalho foi desenvolvido um iii sistema, capaz de navegar com o uso de um mapa topológico global, capaz de se deslocar em um ambiente interno evitando colisões, e capaz de detectar a presença de seres humanos (intrusos) no ambiente. O sistema proposto foi testado em situações reais com o uso de um robô móvel Pioneer P3AT equipado com os sensores Kinect e com uma Câmera FLIR, realizando as tarefas de navegação definidas com sucesso. Outras funcionalidades foram implementadas, como o acompanhamento da pessoa (follow me) e o reconhecimento de comandos gestuais, onde a integração destes módulos com o sistema desenvolvido constituem-se de trabalhos futuros propostos / Mobile robots and service robots are increasing their applications and importance in our modern society. An important type of autonomous mobile robot application is indoor monitoring and surveillance tasks. The adoption of mobile robots for indoor surveillance tasks allows the execution of repetitive environment patrolling, which may even pose risks to the physical integrity of persons. Thus these activities can be autonomously and safely performed by security robots. This work aimed at the development of key modules and components that integrates the general architecture of a surveillance robotic system, including: (i) the development and application of a 3D perception sensor (Kinect) and a thermal sensor (FLIR camera), representing a relatively low-cost solution for mobile robot platforms; (ii) the intruder detection (people) in the environment, through the joint use of 3D and thermal sensors; (iii) the autonomous navigation of mobile robots within obstacle detection and avoidance, performing the monitoring and surveillance tasks of indoor environments; (iv) the identification and recognition of environmental features that allow the robot to perform a navigation based on topological maps. We used methods from Computer Vision, Image Processing and Computational Intelligence to carry out the implementation of the mobile robot surveillance modules. The proximity and distance measurement sensor adopted in the robotic perception system was the Kinect, allowing navigation, obstacle avoidance, and identifying key positions of the robot with respect to a topological map. For the intruder detection task we used a Kinect sensor together with a FLIR thermal camera, integrating the data obtained from both sensors, and thus allowing a better understanding of the environment, and also allowing a greater reliability in people detection. As a main result of this work, it has been v developed a system capable of navigating using a global topological map, capable of moving itself autonomously into an indoor environment avoiding collisions, and capable of detect the presence of humans (intruders) into the environment. The proposed system has been tested in real situations with the use of a Pioneer P3AT mobile robot equipped with Kinect and FLIR camera sensors, performing successfully the defined navigation tasks. Other features have also been implemented, such as following a person and recognizing gestures, proposed as future works to be integrated into the developed system

A análise de comportamento coletivo e rastreamento de pedestres apresentam diversas aplicações, especialmente em sistemas de vigilância inteligente. Neste trabalho é proposta uma solução compreensiva com objetivo de atingir rastreamento de pedestre e reconhecimento de atividade coletiva de maneira robusta baseada na utilização de câmeras calibradas. Primeiramente, com o objetivo de remover a necessidade de calibração manual, nós apresentamos um método de calibração automática que explora detectores de pedestres e remoção de fundo para calibragem baseada em otimização não-linear. Adicionalmente, nós propomos a utilização da matriz de calibração para gerar candidatos coerentes com a geometria de cena em detectores de pedestres. Nossa abordagem tem como objetivo diminuir o intervalo de escalas comumente utilizado em detectores baseados em janelas deslizantes, gerando um número menor de extrações de atributos e reduzindo o número de falsos positivos na detecção. Em seguida, nós propomos um método de rastreamento de múltiplos pedestres utilizando câmeras calibradas. Nossa abordagem explora histogramas de cor para rastrear os pequenas regiões (patches) de cada alvo. Os vetores de deslocamento obtidos através do pareamento de atributos de aparência são combinados com um vetor obtido através de um preditor de movimento em coordenadas de mundo. Adicionalmente, nós incluímos informações originárias de detectores de pedestres para aumentar a acurácia do sistema e sua habilidade de recuperação a falhas. Por fim, nós propomos uma abordagem hierárquica de duas camadas para o problema de reconhecimento de atividade coletiva baseada no uso de classificadores Random Forests. No primeiro nível da técnica proposta, nós utilizamos distâncias entre pares de pessoas e suas respectivas velocidades relativas para classificar interações de pares. Estas interações são combinadas com a dinâmica do formato do grupo observado (e sua respectiva velocidade) para o reconhecimento de atividades coletivas. Os experimentos realizados neste trabalho demonstram a qualidade de nossas abordagens em sequências de vídeos disponíveis publicamente. Nossos resultados mostram serem competitivos quando comparados com técnicas do estado da arte e, particularmente, apresentam uma boa generalização entre diferentes cenários de captura de vídeo. / Collective behavior detection and pedestrian tracking present many applications, specially in surveillance systems. In this dissertation, we proposed a complete pipeline for achieving robust tracking and collective behavior recognition based on calibrated static cameras. To remove the necessity of manual calibration, we first present a fully automatic self-calibration system that explores pedestrian detection results and background removal at non-consecutive frames in order to calibrate a static camera using a non-linear cost function. We also propose the use of camera calibration to generate geometrically coherent candidates for pedestrian detection. Our approach aims to reduce the scale range typically used in sliding-window techniques, which leads to less feature extractions and decreased number of false positives. Then, we propose a multi-target pedestrian tracking algorithm using a calibrated static camera. The tracking approach explores color histograms to track patches of each target. Obtained displacement vectors are combined with the expected motion of pedestrians in the world coordinate system. The proposed tracker also incorporates pedestrian detector results to improve the system’s accuracy and its ability to recover from failure. Finally, we propose a two-layered approach for collective behavior recognition based on Random Forests classifiers. In the first level, we use inter-personal distances and relative speeds computed in the world coordinate system to classify asymmetrical pair interactions. Those interactions are combined with group shape dynamics and mean velocity to recognize the collective behavior. We devise a set of experiments to attest the quality of our approaches using publicly available datasets. Results have shown to be competitive against state-of-the-art techniques, and particularly of good generalization across different databases.