Cotton crop condition was assessed from an analysis of multispectral aerial video imagery. Visible-near infrared imagery of two cotton fields was collected towards the end of the 1990 crop. The digital analysis was based on image classification, and the accuracies were assessed using the Kappa coefficient of agreement. The earliest of three images proved to be best for distinguishing plant variety. Vegetation index images were better for estimating potential yield than the original multispectral image; so too were multi-channel images that were transformed using vegetation indices or principal component analysis. The seedbed preparation rig used, the nitrogen application rate and three plant varieties, a weed species and two cotton cultivars, could all be discriminated from the imagery. Accuracies were moderate for the discrimination of plant variety, tillage treatment and nitrogen treatment, and low for the estimation of potential yield.

Object trajectory tracking is an important topic in many different areas. It is widely used in robot technology, traffic, movie industry, and others. Optical flow is a useful method in the object tracking branch and it can calculate the motion of each pixel between two frames, and thus it provides a possible way to get the trajectory of objects. There are numerous papers describing the implementation of optical flow. Some results are acceptable, but in many projects, there are limitations. In most previous applications, because the camera is usually static, it is easy to apply optical flow to identify the moving targets in a scene and get their trajectories. When the camera moves, a global motion will be added to the local motion, which complicates the issue. In this thesis we use a combination of optical flow and image correlation to deal with this problem, and have good experimental results. For trajectory estimation, we incorporate a Kalman Filter with the optical flow. Not only can we smooth the motion history, but we can also estimate the motion into the next frame. The addition of a spatial-temporal filter improves the results in our later process.

Small Unmanned Aerial Vehicles (SUAVs) are an attractive choice for many surveillance tasks. However, video from an SUAV can be difficult to use in its raw form. In addition, the limitations inherent in the SUAV platform inhibit the distribution of video to remote users. To solve the problems with using SUAV video, we propose a system to automatically create geo-referenced mosiacs of video frames. We also present three novel techniques we have developed to improve ortho-rectification and geo-location accuracy of the mosaics. The most successful of these techniques is able to reduce geo-location error by a factor of 15 with minimal computational overhead. The proposed system overcomes communications limitations by transmitting the mosaics to a central server where there they can easily be accessed by remote users via the Internet. Using flight test results, we show that the proposed mosaicking system achieves real-time performance and produces high-quality and accurately geo-referenced imagery.

Miniature Aerial Vehicles (MAVs) are quickly gaining acceptance as a platform for performing remote sensing or surveillance of remote areas. However, because MAVs are typically flown close to the ground (1000 feet or less in altitude), their field of view for any one image is relatively small. In addition, the context of the video (where and at what orientation are the objects being observed, the relationship between images) is unclear from any one image. To overcome these problems, we propose a geo-referenced mosaicing method that creates a mosaic from the captured images and geo-references the mosaic using information from the MAV IMU/GPS unit. Our method utilizes bundle adjustment within a constrained optimization framework and topology refinement. Using real MAV video, we have demonstrated our mosaic creation process on over 900 frames. Our method has been shown to produce the high quality mosaics to within 7m using tightly synchronized MAV telemetry data and to within 30m using only GPS information (i.e. no roll and pitch information).

This dissertation describes two distinctive methods for human-vehicle interaction recognition: one for ground level videos and the other for aerial videos. For ground level videos, this dissertation presents a novel methodology which is able to estimate a detailed status of a scene involving multiple humans and vehicles. The system tracks their configuration even when they are performing complex interactions with severe occlusion such as when four persons are exiting a car together. The motivation is to identify the 3-D states of vehicles (e.g. status of doors), their relations with persons, which is necessary to analyze complex human-vehicle interactions (e.g. breaking into or stealing a vehicle), and the motion of humans and car doors to detect atomic human-vehicle interactions. A probabilistic algorithm has been designed to track humans and analyze their dynamic relationships with vehicles using a dynamic context. We have focused on two ideas. One is that many simple events can be detected based on a low-level analysis, and these detected events must contextually meet with human/vehicle status tracking results. The other is that the motion clue interferes with states in the current and future frames, and analyzing the motion is critical to detect such simple events. Our approach updates the probability of a person (or a vehicle) having a particular state based on these basic observed events. The probabilistic inference is made for the tracking process to match event-based evidence and motion-based evidence. For aerial videos, the object resolution is low, the visual cues are vague, and the detection and tracking of objects is less reliable as a consequence. Any method that requires accurate tracking of objects or the exact matching of event definition are better avoided. To address these issues, we present a temporal logic based approach which does not require training from event examples. At the low-level, we employ dynamic programming to perform fast model fitting between the tracked vehicle and the rendered 3-D vehicle models. At the semantic-level, given the localized event region of interest (ROI), we verify the time series of human-vehicle relationships with the pre-specified event definitions in a piecewise fashion. With special interest in recognizing a person getting into and out of a vehicle, we have tested our method on a subset of the VIRAT Aerial Video dataset and achieved superior results. / text

Les activités basées sur l'exploitation de données vidéo se sont développées de manière fulgurante ces dernières années : nous assisté à une démocratisation de certaines de ces activités (vidéo-surveillance) mais également à une diversification importante des applications opérationnelles (suivi de ressources naturelles, reconnaissance etc). Cependant, le volume de données vidéo généré est aujourd'hui astronomique et l'efficacité de ces activités est limitée par le coût et la durée nécessaire à l'interprétation humaine des données vidéo. L'analyse automatique de flux vidéos est donc devenue une problématique cruciale pour de nombreuses applications. L'approche semi-automatique développée dans le cadre de cette thèse se concentre plus spécifiquement sur l'analyse de vidéos aériennes, et permet d'assister l'analyste image dans sa tâche en suggérant des zones d'intérêt potentiel par détection de changements. Pour cela, nous effectuons une modélisation tridimensionnelle des apparences observées dans les vidéos de référence. Cette modélisation permet ensuite d'effectuer une détection en ligne des changements significatifs dans une nouvelle vidéo, en identifiant les déviations d'apparence par rapport aux modèles de référence. Des techniques spécifiques ont également été proposées pour effectuer l'estimation des paramètres d'acquisition ainsi que l'atténuation des effets de l'illumination. De plus, nous avons développé plusieurs techniques de consolidation permettant d'exploiter la connaissance a priori relative aux changements à détecter. L'intérêt et les bonnes performances de notre approche a été minutieusement démontré à l'aide de données réelles et synthétiques. / Business activities based on the use of video data have developed at a dazzling speed these last few years: not only has the market of some of these activities widely expanded (video-surveillance) but the operational applications have also greatly diversified (natural resources monitoring, intelligence etc). However, nowadays, the volume of generated data has become overwhelming and the efficiency of these activities is now limited by the cost and the time required by the human interpretation of this video data. Automatic analysis of video streams has hence become a critical problem for numerous applications. The semi-autmoatic approach developed in this thesis focuses more specifically on the automatic analysis of aerial videos and enables assisting the image analyst in his task by suggesting areas of potential interest identified using change detection. For that purpose, our approach proceeds to a tridimensional modeling of the appearances observed in the reference videos. Such a modeling then enables the online detection of significant changes in a new video, by identifying appearance deviations with respect to the reference models. Specific techniques have also been developed to estimate the acquisition parameters and to attenuate illumination effects. Moreover, we developed several consolidation techniques making use of a priori knowledge related to targeted changes, in order to improve detection accuracy. The interest and good performance of our change detection approach has been carefully demonstrated using both real and synthetical data.

In this dissertation, we address the problem of detecting humans and vehicles, tracking them in crowded scenes, and finally determining their activities in aerial video. Even though this is a well explored problem in the field of computer vision, many challenges still remain when one is presented with realistic data. These challenges include large camera motion, strong scene parallax, fast object motion, large object density, strong shadows, and insufficiently large action datasets. Therefore, we propose a number of novel methods based on exploiting scene constraints from the imagery itself to aid in the detection and tracking of objects. We show, via experiments on several datasets, that superior performance is achieved with the use of proposed constraints. First, we tackle the problem of detecting moving, as well as stationary, objects in scenes that contain parallax and shadows. We do this on both regular aerial video, as well as the new and challenging domain of wide area surveillance. This problem poses several challenges: large camera motion, strong parallax, large number of moving objects, small number of pixels on target, single channel data, and low frame-rate of video. We propose a method for detecting moving and stationary objects that overcomes these challenges, and evaluate it on CLIF and VIVID datasets. In order to find moving objects, we use median background modelling which requires few frames to obtain a workable model, and is very robust when there is a large number of moving objects in the scene while the model is being constructed. We then iii remove false detections from parallax and registration errors using gradient information from the background image. Relying merely on motion to detect objects in aerial video may not be sufficient to provide complete information about the observed scene. First of all, objects that are permanently stationary may be of interest as well, for example to determine how long a particular vehicle has been parked at a certain location. Secondly, moving vehicles that are being tracked through the scene may sometimes stop and remain stationary at traffic lights and railroad crossings. These prolonged periods of non-motion make it very difficult for the tracker to maintain the identities of the vehicles. Therefore, there is a clear need for a method that can detect stationary pedestrians and vehicles in UAV imagery. This is a challenging problem due to small number of pixels on the target, which makes it difficult to distinguish objects from background clutter, and results in a much larger search space. We propose a method for constraining the search based on a number of geometric constraints obtained from the metadata. Specifically, we obtain the orientation of the ground plane normal, the orientation of the shadows cast by out of plane objects in the scene, and the relationship between object heights and the size of their corresponding shadows. We utilize the above information in a geometry-based shadow and ground plane normal blob detector, which provides an initial estimation for the locations of shadow casting out of plane (SCOOP) objects in the scene. These SCOOP candidate locations are then classified as either human or clutter using a combination of wavelet features, and a Support Vector Machine. Additionally, we combine regular SCOOP and inverted SCOOP candidates to obtain vehicle candidates. We show impressive results on sequences from VIVID and CLIF datasets, and provide comparative quantitative and qualitative analysis. We also show that we can extend the SCOOP detection method to automatically estimate the iv orientation of the shadow in the image without relying on metadata. This is useful in cases where metadata is either unavailable or erroneous. Simply detecting objects in every frame does not provide sufficient understanding of the nature of their existence in the scene. It may be necessary to know how the objects have travelled through the scene over time and which areas they have visited. Hence, there is a need to maintain the identities of the objects across different time instances. The task of object tracking can be very challenging in videos that have low frame rate, high density, and a very large number of objects, as is the case in the WAAS data. Therefore, we propose a novel method for tracking a large number of densely moving objects in an aerial video. In order to keep the complexity of the tracking problem manageable when dealing with a large number of objects, we divide the scene into grid cells, solve the tracking problem optimally within each cell using bipartite graph matching and then link the tracks across the cells. Besides tractability, grid cells also allow us to define a set of local scene constraints, such as road orientation and object context. We use these constraints as part of cost function to solve the tracking problem; This allows us to track fast-moving objects in low frame rate videos. In addition to moving through the scene, the humans that are present may be performing individual actions that should be detected and recognized by the system. A number of different approaches exist for action recognition in both aerial and ground level video. One of the requirements for the majority of these approaches is the existence of a sizeable dataset of examples of a particular action from which a model of the action can be constructed. Such a luxury is not always possible in aerial scenarios since it may be difficult to fly a large number of missions to observe a particular event multiple times. Therefore, we propose a method for v recognizing human actions in aerial video from as few examples as possible (a single example in the extreme case). We use the bag of words action representation and a 1vsAll multi-class classification framework. We assume that most of the classes have many examples, and construct Support Vector Machine models for each class. Then, we use Support Vector Machines that were trained for classes with many examples to improve the decision function of the Support Vector Machine that was trained using few examples, via late weighted fusion of decision values.

Les travaux réalisés dans le cadre de cette thèse ont pour objectif d’extraire les différents segments informatifs au sein de séquences vidéo, plus particulièrement aériennes. L’interprétation manuelle de telles vidéos dans une optique de renseignement se heurte en effet au volume des données disponibles. Une assistance algorithmique fondée sur diverses modalités d’indexation est donc envisagée, dans l’objectif de repérer les "segments d’intérêt" et éviter un parcours intégral de la vidéo. Deux approches particulières ont été retenues et respectivement développées au sein de chaque partie. La partie 1 propose une utilisation des conditions de prise de vue (CPDV) comme modalités d’indexation. Une évaluation de la qualité image permet ainsi de filtrer les segments temporels de mauvaise qualité et donc inexploitables. La classification du mouvement image apparent directement lié au mouvement caméra, fournit une indexation de séquences vidéo en soulignant notamment les segments potentiels d’intérêt ou au contraire les segments difficiles présentant un mouvement très rapide ou oscillant. La partie 2 explore le contenu dynamique de la séquence vidéo, plus précisément la présence d’objets en mouvement. Une première approche locale en temps est présentée. Elle filtre les résultats d’une première classification par apprentissage supervisé en exploitant les informations de contexte, spatial puis sémantique. Différentes approches globales en temps sont par la suite explorées. De telles approches permettent de garantir la cohérence temporelle des résultats et réduire les fausses alarmes. / The objective of this thesis is to extract the informative temporal segments from video sequences, more particularly in aerial video. Manual interpretation of such videos for information gathering faces an ever growing volume of available data. We have thus considered an algorithmic assistance based on different modalities of indexation in order to locate "segments of interest" and avoid a complete visualization of the video. We have chosen two methods in particular and have respectively developed them in each part of this thesis. Part 1 describes how viewing conditions can be used as a method of indexation. The assessment of image quality enables to filter out the temporal segments for which the quality is low and which can thus not be exploited. The classification of global image motion, which is directly linked to camera motion, leads to a method of indexation for video sequences. Indeed, it emphasizes possible segments of interest or, conversely, difficult segments for which motion is very fast or oscillating. Part 2 focuses on the dynamic content of video sequences, especially the presence of moving objects. We first present a local (in time) approach. This approach refines the results obtained after a first classification by supervised learning by using contextual information, spatial then semantic information. We have then investigated several methods for moving object detection which are global in time. Such approaches aim to enforce the temporal consistency of the detected objects and to reduce false detections.

This thesis proposes a system for extraction of vehicle trajectories from aerial video data for traffic analysis. The system is designed to analyse video sequence of a single traffic scene captured by an action camera mounted on an arbitrary UAV flying at the altitudes of approximately 150 m. Each video frame is geo-registered using visual correspondence of extracted ORB features. For the detection of vehicles, MB-LBP classifier cascade is deployed, with additional step of pre-filtering of detection candidates based on movement and scene context. Multi-object tracking is achieved by Bayesian bootstrap filter with an aid of the detection algorithm. The performance of the system was evaluated on three extensively annotated datasets. The results show that on the average, 92% of all extracted trajectories are corresponding to the reality. The system is already being used in the research to aid the process of design and analysis of road infrastructures.

Video acquired using a camera mounted on a mini Unmanned Air Vehicle (mUAV) may be very helpful in Wilderness Search and Rescue and many other applications but is commonly plagued with limited spatial and temporal field of views, distractive jittery motions, disorienting rotations, and noisy and distorted images. These problems collectively make it very difficult for human viewers to identify objects of interest as well as infer correct orientations throughout the video. In order to expand the temporal and spatial field of view, stabilize, and better orient users of noisy and distorted mUAV video, a method is proposed of estimating in software and in real time the relative motions of each frame to the next by tracking a small subset of features within each frame to the next. Using these relative motions, a local Euclidean mosaic of the video can be created and a curve can be fit to the video's accumulative motion path to stabilize the presentations of both the video and the local Euclidean mosaic. The increase in users' abilities to perform common search-and-rescue tasks of identifying objects of interest throughout the stabilized and locally mosaiced mUAV video is then evaluated. Finally, a discussion of remaining limitations is presented along with some possibilities for future work.