Many tasks require surveillance and analysis in order to make decisions regarding the next course of action. The people responsible for these tasks are usually concerned with any event that affects their bottom-line. Traditionally, human operators have had to either actively man a set of video displays to determine if specific events were occurring or manually review hours of collected video data to see if a specific event occurred. Actively monitoring video stream or manually reviewing and analyzing the data collected, however, is a tedious and long process which is prone to errors due to biases and inattention. Automatically processing and analyzing the video provides an alternate way of getting more accurate results because it can reduce the likelihood of missing important events and the human factors that lead to decreased efficiency. The thesis aims to contribute to the area of using computer vision as a decision support tool by integrating detector, tracker, re-identification, activity status estimation, and event processor modules to generate the necessary event statistics needed by a human operator. The contribution of this thesis is a system that uses feedback from each of the modules to provide better target detection, and tracking results for event statistics generation over an extended period of time. To demonstrate the efficacy of the proposed system, it is first used to generate event statistics that measure productivity on multiple construction work sites. The versatility of the proposed system is also demonstrated in an indoor assisted living environment by using it to determine how much of an influence a technology intervention had on promoting interactions amongst older adults in a shared space.

In recent years unmanned aircraft systems (UAS) have been successfully used in a wide variety of applications. Their value as surveillance platforms has been proven repeatedly in both military and civilian domains. As substitutes to human inhabited aircraft, they fulfill missions that are dull, dirty and dangerous. Representative examples of successful use of UAS are in areas including battlefield assessment, reconnaissance, port security, wildlife protection, wildfire detection, search and rescue missions, border security, resource exploration and oil spill detection. The reliance of almost every UAS application on the ability to sense, detect, see and avoid from a distance has motivated this thesis, attempting to further investigate this issue. In particular, among the various types of UAS, small scale unmanned rotorcraft or Vertically Take-off and Landing, (VTOL) vehicles have been chosen to serve as the sensor carrier platforms because of their operational flexibility. In this work we address the problem of object identification and tracking in a largely unknown dynamic environment under the additional constraint of real-time operation and limited computational power. In brief, the scope of this thesis can be stated as follows: Design a vision system for a small autonomous helicopter that will be able to: Identify arbitrary objects using a minimal description model and a-priori knowledge; Track objects of interest; Operate in real-time; Operate in a largely unknown, dynamically changing, outdoors environment under the following constraints: Limited processing power and payload; Low cost, off-the-shelf components. The main design directives remain that of real-time execution and low price, high availability components. It is in a sense an investigation for the minimum required hardware and algorithmic complexity to accomplish the desired tasks. After development, the system was evaluated as to its suitability in an array of applications. The ones that were chosen for that purpose were: Detection of semi-concealed objects; Detection of a group of ground robots; Traffic monitoring. Adequate performance was demonstrated in all of the above cases.

Abnormal behaviour may indicate important objects and events in a wide variety of domains. One such domain is intelligence and surveillance, where there is a clear trend towards more and more advanced sensor systems producing huge amounts of trajectory data from moving objects, such as people, vehicles, vessels and aircraft. In the maritime domain, for example, abnormal vessel behaviour, such as unexpected stops, deviations from standard routes, speeding, traffic direction violations etc., may indicate threats and dangers related to smuggling, sea drunkenness, collisions, grounding, hijacking, piracy etc. Timely detection of these relatively infrequent events, which is critical for enabling proactive measures, requires constant analysis of all trajectories; this is typically a great challenge to human analysts due to information overload, fatigue and inattention. In the Baltic Sea, for example, there are typically 3000–4000 commercial vessels present that are monitored by only a few human analysts. Thus, there is a need for automated detection of abnormal trajectory patterns. In this thesis, we investigate algorithms appropriate for automated detection of anomalous trajectories in surveillance applications. We identify and discuss some key theoretical properties of such algorithms, which have not been fully addressed in previous work: sequential anomaly detection in incomplete trajectories, continuous learning based on new data requiring no or limited human feedback, a minimum of parameters and a low and well-calibrated false alarm rate. A number of algorithms based on statistical methods and nearest neighbour methods are proposed that address some or all of these key properties. In particular, a novel algorithm known as the Similarity-based Nearest Neighbour Conformal Anomaly Detector (SNN-CAD) is proposed. This algorithm is based on the theory of Conformal prediction and is unique in the sense that it addresses all of the key properties above. The proposed algorithms are evaluated on real world trajectory data sets, including vessel traffic data, which have been complemented with simulated anomalous data. The experiments demonstrate the type of anomalous behaviour that can be detected at a low overall alarm rate. Quantitative results for learning and classification performance of the algorithms are compared. In particular, results from reproduced experiments on public data sets show that SNN-CAD, combined with Hausdorff distance  for measuring dissimilarity between trajectories, achieves excellent classification performance without any parameter tuning. It is concluded that SNN-CAD, due to its general and parameter-light design, is applicable in virtually any anomaly detection application. Directions for future work include investigating sensitivity to noisy data, and investigating long-term learning strategies, which address issues related to changing behaviour patterns and increasing size and complexity of training data.

In the recent years, connected surveillance systems have been witnessing an unprecedented evolution owing to the advancements in internet of things and deep learning technologies. However, vulnerabilities to various kinds of attacks both at the cyber network-level and at the physical worldlevel are also rising. This poses danger not only to the devices but also to human life and property. The goal of this thesis is to enhance the security of an internet of things, focusing on connected video-based surveillance systems, by proposing multiple novel solutions to address security issues at the cyber network-level and to defend such systems at the physical world-level. In order to enhance security at the cyber network-level, this thesis designs and develops solutions to detect network intrusions in an internet of things such as surveillance cameras. The first solution is a novel method for network flow features transformation, named TempoCode. It introduces a temporal codebook-based encoding of flow features based on capturing the key patterns of benign traffic in a learnt temporal codebook. The second solution takes an unsupervised learning-based approach and proposes four methods to build efficient and adaptive ensembles of neural networks-based autoencoders for intrusion detection in internet of things such as surveillance cameras. To address the physical world-level attacks, this thesis studies, for the first time to the best of our knowledge, adversarial patches-based attacks against a convolutional neural network (CNN)- based surveillance system designed for vehicle make and model recognition (VMMR). The connected video-based surveillance systems that are based on deep learning models such as CNNs are highly vulnerable to adversarial machine learning-based attacks that could trick and fool the surveillance systems. In addition, this thesis proposes and evaluates a lightweight defense solution called SIHFR to mitigate the impact of such adversarial-patches on CNN-based VMMR systems, leveraging the symmetry in vehicles’ face images. The experimental evaluations on recent realistic intrusion detection datasets prove the effectiveness of the developed solutions, in comparison to state-of-the-art, in detecting intrusions of various types and for different devices. Moreover, using a real-world surveillance dataset, we demonstrate the effectiveness of the SIHFR defense method which does not require re-training of the target VMMR model and adds only a minimal overhead. The solutions designed and developed in this thesis shall pave the way forward for future studies to develop efficient intrusion detection systems and adversarial attacks mitigation methods for connected surveillance systems such as VMMR.