Natural disasters damage infrastructure and create significant obstacles to humanitarian aid efforts. Roads may become unusable, hindering or halting efforts to provide food, water, shelter, and life-saving emergency care. Finding a safe route during a disaster is especially difficult because as the disaster unfolds, the usability of roads and other infrastructure can change quickly, rendering most navigation services useless. With the proliferation of cheap cameras and unmanned aerial vehicles [UAVs], the rapid collection of aerial data after a natural disaster has become increasingly common. This data can be used to quickly appraise the damage to critical infrastructure, which can help solve navigational and logistical problems that may arise after the disaster. This work focuses on a framework in which a UAV is paired with an unmanned ground vehicle [UGV]. The UAV follows the UGV with a downward-facing camera and helps the ground vehicle navigate the flooded environment. This work makes several contributions: a simulation environment is created to allow for automated data collection in hypothetical disaster scenarios. The simulation environment uses real-world satellite and elevation data to emulate natural disasters such as floods. The environment partially simulates the dynamics of the UAV and UGV, allowing agents to ex- plore during hypothetical disasters. Several semantic image segmentation models are tested for efficacy in identifying obstacles and creating cost maps for navigation within the environ- ment, as seen by the UAV. A deep homography model incorporates temporal relations across video frames to stitch cost maps together. A weighted version of a navigation algorithm is presented to plan a path through the environment. The synthesis of these modules leads to a novel framework wherein a UAV may guide a UGV safely through a disaster area. / Master of Science / Damage to infrastructure after a natural disaster can make navigation a major challenge. Imagine a hurricane has hit someone's house; they are hurt and need to go to the hospital. Using a traditional GPS navigation system or even their memory may not work as many roads could be impassible. However, if the GPS could be quickly updated as to which roads were not flooded, it could still be used to navigate and avoid hazards. While the system presented is designed to work with a self-driving vehicle, it could easily be extended to give directions to a human. The goal of this work is to provide a system that could be used as a replacement for a GPS based on aerial photography. The advantage of this system is that flooded or damaged infrastructure can be identified and avoided in real-time. The system could even identify other possible routes by using photography, such as driving across a field to reach higher ground. Like a GPS, the system works automatically, tracking a user's position and sug- gesting turns, aiding navigation. A contribution of this work is a simulation of the environment designed in a video game engine. The game engine creates a video game world that can be flooded and used to test the new navigation system. The video game environment is used to train an artificial intel- ligence computer model to identify hazards and create routes that would avoid them. The system could be used in a real-world disaster following training in a video game world.

Precision agriculture allows farmers to efficiently use their resources with site-specific applications. The current work looks to computer vision for the data collection method necessary for such a smart field, including cameras sensitive to visual (430-650~nm), near infrared (NIR,750-900~nm), shortwave infrared (SWIR,950-1700~nm), and longwave infrared (LWIR,7500-16000~nm) light. Three areas are considered in the study: image segmentation, multispectral image registration, and the feature tracking of a stressed plant. The accuracy of several image segmentation methods are compared. Basic thresholding on pixel intensities and vegetation indices result in accuracies below 75% . Neural networks (NNs) and support vector machines (SVMs) label correctly at 89% and 79%, respectively, when given only visual information, and final accuracies of 97% when the near infrared is added. The point matching methods of Scale Invariant Feature Transform (SIFT) and Edge Orient Histogram (EOH) are compared for accuracy. EOH improves the matching accuracy, but ultimately not enough for the current work. In order to track the image features of a stressed plant, a set of basil and catmint seedlings are grown and placed under drought and hypoxia conditions. Trends are shown in the average pixel values over the lives of the plants and with the vegetation indices, especially that of Marchant and NIR. Lastly, trends are seen in the image textures of the plants through use of textons. / Master of Science

With vast improvements in computing power in the last two decades, humans have invested significantly in engineering resources in an attempt to automate labor intensive or dangerous tasks. A particularly dangerous and labor-intensive task is painting lines on roads for facilitating urban mobility. This thesis proposes an approach to automate the process of painting lines on the ground using an autonomous ground vehicle (AGV) fitted with a stabilized painting mechanism. The AGV accepts Global Positioning System (GPS) coordinates for waypoint navigation. A computer vision algorithm is developed to provide vision feedback to stabilize the painting mechanism. The system is demonstrated to follow an input desired trajectory and cancel any high frequency vibrations due to the uneven terrain that the vehicle is traversing. Also, the stabilizing system is able to eliminate the long-term drift (due to inaccurate GPS waypoint navigation) using the complementary vision system. / MS / There is a need to develop an automated system capable of painting lines on the ground with minimal human intervention as the current methods to paint lines on the ground are inefficient, labor intensive, and dangerous. The human input to such a system is limited to the determination of the desired trajectory of the line to be drawn. This thesis presents the design and development of an autonomous line painting system that includes an autonomous ground vehicle (capable of following GPS waypoints) integrated with an automatic line painting mechanism. As the vehicle traverses the ground, it experiences disturbances due to the interaction between the wheels and the ground, and also a long-term drift due to inaccurate tracking of the input GPS coordinates. In order to compensate for these disturbances, a vision system is implemented providing feedback to a stabilizing arm. This automated system is able to demonstrate the capability to follow a square trajectory defined by GPS coordinates while compensating for the disturbances.

Most neural networks use a normal convolutional layer that assumes that all input pixels are valid pixels. However, pixels added to the input through padding result in adding extra information that was not initially present. This extra information can be considered invalid. Invalid pixels can also be inside the image where they are referred to as holes in completion tasks like image inpainting. In this work, we look for a method that can handle both types of invalid pixels. We compare on the same test bench two methods previously used to handle invalid pixels outside the image (Partial and Edge convolutions) and one method that was designed for invalid pixels inside the image (Gated convolution). We show that Partial convolution performs the best in image classification while Gated convolution has the advantage on semantic segmentation. As for hotel recognition with masked regions, none of the methods seem appropriate to generate embeddings that leverage the masked regions. / Master of Science / A module at the heart of deep neural networks built for Artificial Intelligence is the convolutional layer. When multiple convolutional layers are used together with other modules, a Convolutional Neural Network (CNN) is obtained. These CNNs can be used for tasks such as image classification where they tell if the object in an image is a chair or a car, for example. Most CNNs use a normal convolutional layer that assumes that all parts of the image fed to the network are valid. However, most models zero pad the image at the beginning to maintain a certain output shape. Zero padding is equivalent to adding a black frame around the image. These added pixels result in adding information that was not initially present. Therefore, this extra information can be considered invalid. Invalid pixels can also be inside the image where they are referred to as holes in completion tasks like image inpainting where the network is asked to fill these holes and give a realistic image. In this work, we look for a method that can handle both types of invalid pixels. We compare on the same test bench two methods previously used to handle invalid pixels outside the image (Partial and Edge convolutions) and one method that was designed for invalid pixels inside the image (Gated convolution). We show that Partial convolution performs the best in image classification while Gated convolution has the advantage on semantic segmentation. As for hotel recognition with masked regions, none of the methods seem appropriate to generate embeddings that leverage the masked regions.

With the rapid proliferation of small unmanned aircraft systems (UAS), the risk of mid-air collisions is growing, as is the risk associated with the malicious use of these systems. The airborne detect-and-avoid (ABDAA) problem and the counter-UAS problem have similar sensing requirements for detecting and tracking airborne threats. In this dissertation, two image-based sensing methods are merged to mimic human vision in support of counter-UAS applications. In the proposed sensing system architecture, a ``peripheral vision'' camera (with a fisheye lens) provides a large field-of-view while a ``central vision'' camera (with a perspective lens) provides high resolution imagery of a specific object. This pair form a heterogeneous stereo vision system that can support range resolution. A novel peripheral-central vision (PCV) system to detect, localize, and classify an airborne threat is first introduced. To improve the developed PCV system's capability, three novel algorithms for the PCV system are devised: a model-based path prediction algorithm for fixed-wing unmanned aircraft, a multiple threat scheduling algorithm considering not only the risk of threats but also the time required for observation, and the heterogeneous stereo-vision optimal placement (HSOP) algorithm providing optimal locations for multiple PCV systems to minimize the localization error of threat aircraft. The performance of algorithms is assessed using an experimental data set and simulations. / Doctor of Philosophy / With the rapid proliferation of small unmanned aircraft systems (UAS), the risk of mid-air collisions is growing, as is the risk associated with the malicious use of these systems. The sensing technologies for detecting and tracking airborne threats have been developed to solve these UAS-related problems. In this dissertation, two image-based sensing methods are merged to mimic human vision in support of counter-UAS applications. In the proposed sensing system architecture, a ``peripheral vision'' camera (with a fisheye lens) provides a large field-of-view while a ``central vision'' camera (with a perspective lens) provides high resolution imagery of a specific object. This pair enables estimation of an object location using the different viewpoints of the different cameras (denoted as ``heterogeneous stereo vision.'') A novel peripheral-central vision (PCV) system to detect an airborne threat, estimate the location of the threat, and determine the threat class (e.g. aircraft, bird) is first introduced. To improve the developed PCV system's capability, three novel algorithms for the PCV system are devised: an algorithm to predict the future path of an fixed-wing unmanned aircraft, an algorithm to decide an efficient observation schedule for multiple threats, and an algorithm that provides optimal locations for multiple PCV systems to estimate the threat position better. The performance of algorithms is assessed using an experimental data set and simulations.

This metaverse and computer vision-powered application is designed to serve people with low vision or a visual impairment, ranging from adults to old age. Specifically, we hope to improve the situational awareness of users in a scene by narrating the visual content from their point of view. The user would be able to understand the information through auditory channels as the system would narrate the scene's description using speech technology. This could increase the accessibility of visual-spatial information for the users in a metaverse and later in the physical world. This solution is designed and developed considering the hypothesis that if we enable the narration of a scene's visual content, we can increase the understanding and access to that scene. This study paves the way for VR technology to be used as a training and exploration tool not limited to blind people in generic environments, but applicable to specific domains such as military, healthcare, or architecture and planning. We have run a user study and evaluated our hypothesis about which set of algorithms will perform better for a specific category of tasks - like search or survey - and evaluated the narration algorithms by the user's ratings of naturalness, correctness and satisfaction. The tasks and algorithms have been discussed in detail in the chapters of this thesis. / Master of Science / The solution is built using an object detection algorithm and virtual environments which run on the web browser using X3DOM. The solution would help improve situational awareness for normal people as well as for low vision individuals through speech. On a broader scale, we seek to contribute to accessibility solutions. We have designed four algorithms which will help user to understand the scene information through auditory channels as the system would narrate the scene's description using speech technology. The idea would increase the accessibility of visual-spatial information for the users in a metaverse and later in the physical world.

The demand and interest in serving Computer Vision applications at the Edge, where Edge Devices generate vast quantities of data, clashes with the reality that many Devices are largely unable to process their data in real time. While computational offloading, not to the Cloud but to nearby Edge Nodes, offers convenient acceleration for these applications, such systems are not without their constraints. As Edge networks may be unreliable or wireless, offloading quality is sensitive to communication bottlenecks. Unlike seemingly unlimited Cloud resources, an Edge Node, serving multiple clients, may incur delays due to resource contention. This project describes relevant Computer Vision workloads and how an effective offloading framework must adapt to the constraints that impact the Quality of Service yet have not been effectively nor properly addressed by previous literature. We design an offloading controller, based on closed-loop control theory, that enables Devices to maximize their throughput by appropriately offloading under variable conditions. This approach ensures a Device can utilize the maximum available offloading bandwidth. Finally, we constructed a realistic testbed and conducted measurements to demonstrate the superiority of our offloading controller over previous techniques. / Master of Science / Devices like security cameras and some Internet of Things gadgets produce valuable real-time video for AI applications. A field within AI research called Computer Vision aims to use this visual data to compute a variety of useful workloads in a way that mimics the human visual system. However, many workloads, such as classifying objects displayed in a video, have large computational demands, especially when we want to keep up with the frame rate of a real-time video. Unfortunately, these devices, called Edge Devices because they are located far from Cloud datacenters at the edge of the network, are notoriously weak for Computer Vision algorithms, and, if running on a battery, will drain it quickly. In order to keep up, we can offload the computation of these algorithms to nearby servers, but we need to keep in mind that the bandwidth of the network might be variable and that too many clients connected to a single server will overload it. A slow network or an overloaded server will incur delays which slow processing throughput. This project describes relevant Computer Vision workloads and how an effective offloading framework that effectively adapts to these constraints has not yet been addressed by previous literature. We designed an offloading controller that measures feedback from the system and adapts how a Device offloads computation, in order to achieve the best possible throughput despite variable conditions. Finally, we constructed a realistic testbed and conducted measurements to demonstrate the superiority of our offloading controller over previous techniques.

The extraction and analysis of human gait characteristics using image sequences and the subsequent classification of these characteristics are currently an intense area of research. Recently, the focus of this research area has turned to the realm of computer vision as an unobtrusive way of performing this analysis. With such systems becoming more common, a gait analysis system that will quickly and accurately determine if a subject is walking normally becomes more valuable. Such a system could be used as a preprocessing step in a more sophisticated gait analysis system or could be used for rehabilitation purposes. In this thesis a system is proposed which utilizes a novel fusion of spatial computer vision operations as well as motion in order to accurately and efficiently determine if a subject moving through a scene is walking normally or abnormally. Specifically this system will yield a classification of the type of motion being observed, whether it is a human walking normally or some other kind of motion taking place within the frame. Experimental results will show that the system provides accurate detection of normal walking and can distinguish abnormalities as subtle as limping or walking with a straight leg reliably. / Master of Science

This thesis describes a mosaicking system designed to generate image mosaics that facilitate size estimation of 3-dimensional objects by improving data obtained with a multi-sensor video camera. The multi-sensor camera is equipped with a pulse laser-rangefinder and internally mounted inclinometers that measure instrument orientation about three axes. Using orientation data and video data, mosaics are constructed to reduce orientation data errors by augmenting orientation data with image information. Mosaicking is modeled as a 7-step refinement process: 1) an initial mosaic is constructed using orientation information obtained from the camera's inclinometers; 2) mosaics are refined by using coarse-to-fine processing to minimize an energy metric and, consequently, align overlapping video frames; 3) pair-wise mosaicking errors are detected, and removed, using an energy-based confidence metric; 4) mosaic accuracy is refined via color analysis; 5) mosaic accuracy is refined by estimating an affine transformation to align overlapping frames; 6) affine transformation approximations between overlapping video frames are used to reduce image noise through super-resolution; 7) original orientation data are corrected given the refined orientations of images within the mosaic. The mosaicking system has been tested using objects of known size and orientation accuracy has been improved by 86% for these cases. / Master of Science

This work focuses on automated classification of railway ties based on their condition using aerial imagery. Four approaches are explored and compared to achieve this goal - handcrafted features, HOG features, transfer learning and proposed CNN architecture. Mean test accuracy per class and Quadratic Weighted Kappa score are used as performance metrics, particularly suited for the ordered classification in this work. Transfer learning approach outperforms the handcrafted features and HOG features by a significant margin. The proposed CNN architecture caters to the unique nature of the railway tie images and their defects. The performance of this approach is superior to the handcrafted and HOG features. It also shows a significant reduction in the number of parameters as compared to the transfer learning approach. Data augmentation boosts the performance of all approaches. The problem of label noise is also analyzed. The techniques proposed in this work will help in reducing the time, cost and dependency on experts involved in traditional railway tie inspections and will facilitate efficient documentation and planning for maintenance of railway ties. / Master of Science / Railway tracks and their components need to be frequently inspected for defects or design non-compliances. Manual inspections involve long hours, high cost and dependency on the availability of experts. Previous efforts to automate the inspection of the railway track inspections are focused towards either other components of the railway track or towards using custom designed ground vehicles. This work presents four approaches to automate the inspection of wooden railway ties by categorizing them into one of three categories based on their condition. Images of the track are taken by an aerial vehicle, in which the track is left untouched. The techniques of computer vision and machine learning used in this work outperform the baselines. The efforts are directed towards making the algorithm learn from the labeled data. The labeled data is also artificially enlarged and enriched, which boosts the performance of the classifiers. The performance metrics used to evaluate the classification approaches are particularly suited for the task at hand. The problem of inconsistency in labeling between two human labelers is analyzed. Potential further directions for research are also identified.