In almost all computer vision and perception based applications, particularly with camera and lidar; state-of-the-art algorithms are all based upon deep neural networks which require large amounts of data. Thus, the ability to label data accurately and quickly is of great importance. Approaches to semi-automated labeling (SAL) thus far have relied on using state-of-the-art object detectors to assist with labeling; however, these approaches still require a significant number of manual corrections. Surprisingly, none of these approaches have considered labeling from the perspective of multiple diverse algorithms. In this thesis a new framework for semi-automated labeling is presented, it is called F-SAL which stands for Fusion Based Semi-automated Labeling. Firstly, F-SAL extends on the idea of SAL through introducing multi-algorithm fusion with learning based feedback. Secondly, it incorporates new stages such as uncertainty evaluation and diversity evaluation. All the algorithms and design choices regarding localization fusion, label fusion, uncertainty and diversity evaluation are presented and discussed in significant detail. The biggest advantage of F-SAL is that through the fusion of algorithms, the number of true detections is either more or equivalent to the best single detector; while the false alarms are suppressed significantly. In the case of a single detector, to lower the false alarm rate, detector parameters must be adjusted, which trade lower false alarms for fewer detections. With F-SAL, a lower false alarm rate can be achieved without sacrificing any detections, as false alarms are suppressed during fusion, and true detections are maximized through diversity. Results on several datasets for image and lidar data show that F-SAL outperforms the single best detector in all scenarios. / Thesis / Master of Applied Science (MASc)

Different architectures of gating networks that aggregate information from multiple modalities and their suitability for decision fusion is investigated. The research question, how does a gating network for decision fusion in multimodal classification problem compare to other alternatives, is answered by a quantitative and inductive reasoning approach. This is done by training different machine learning methods on individual modalities and fusing their predictions forthe final classification using M-MNIST, a new data set with three modalities (image, audio, and text). The gating networks achieve greater classification accuracy when fusing information from all modalities, in contrast to considering only one modality, or without fusion. The gating network potential is demonstrated by training it on modalities with different levels of classification accuracy where it achieves the highest average normalized gain when scoring the highest validation accuracy of the three fusion methods, where the results indicate that the gating network can suppress noise in the data. Moreover, by adding an additional weak modality to the gating network, the classification accuracy is improved, hinting at that there might be an incentive to use many weak modalities instead of a few strong ones. / Olika arkitekturer för gating-nätverk som aggregerar information från flera olika modaliteter undersöks här, liksom deras lämplighet för användning för att förena olika beslutsunderlag. Forskningsfrågan ”Hur bra står sig ett gating- nätverk för att ensa beslutsunderlag i multimodala klassificeringsproblem?” besvaras med ett kvantitativt och induktivt tillvägagångssätt. Olika maskininlärningsmetoder har tränats på singulära modaliteter och sedan ensa deras prediktioner för klassificering i M-MNIST: en ny ansamling data med tre modaliteter (bild, ljud och text). Nätverket uppnår bättre resultat i klassificeringen när information från alla modaliteter används, än när endast en modalitet används (eller utan ensning). Nätverkets potential har kunnat illustreras genom träning på modaliteter med olika nivåer av klassificeringskapacitet. Det får bästa resultat, mätt i högsta genomsnittliga normaliserade ökning, i samband med högsta valideringsresultat av de tre metoderna för ensning. Här indikerar resultaten att gating-nätverket kan undertrycka brus i datat. Genom att lägga till ytterligare en (svag) modalitet till nätverket så kan klassificeringens kvalitet ökas på, vilket antyder att det kan finnas skäl att använda många svaga modaliteter iställer för få starka modaliteter.

Le niveau de complexité croissant des systèmes et les exigences de performances et de sûreté de fonctionnement qui leur sont associées ont induit la nécessité de développer de nouvelles approches de surveillance. Les travaux de cette thèse portent sur la surveillance des systèmes complexes, notamment la détection, le diagnostic et le pronostic de défauts, avec une méthodologie basée sur la fusion de décisions. L’objectif principal est de proposer une approche générique de fusion de diverses méthodes de surveillance, dont la performance serait meilleure que celles des méthodes individuelles la composant. Pour cela, nous avons proposé une nouvelle démarche de fusion de décisions, basée sur la théorie Bayésienne. Cette démarche s’appuie sur une déduction théorique des paramètres du Réseau Bayésien en fonction des objectifs de performance à atteindre en surveillance. Le développement conduit à un problème multi-objectif sous contraintes, résolu par une approche lexicographique. La première étape se déroule hors-ligne et consiste à définir les objectifs de performance à respecter afin d’améliorer les performances globales du système. Les paramètres du réseau Bayésien permettant de respecter ces objectifs sont ensuite déduits de façon théorique. Enfin, le réseau Bayésien paramétré est utilisé en ligne afin de tester les performances de la fusion de décisions. Cette méthodologie est adaptée et appliquée d’une part à la détection et au diagnostic, et d’autre part au pronostic. Les performances sont évaluées en termes de taux de diagnostic de défauts (FDR) et taux de fausses alarmes (FAR) pour l’étape de détection et de diagnostic, et en durée de fonctionnement avant la défaillance du système (RUL) pour le pronostic. / Nowadays, systems are becoming more and more complex and require new effective methods for their supervision. This latter comprises a monitoring phase that aims to improve the system’s performances and ensure a safety production for humans and materials. This thesis work deals with fault detection, diagnosis and prognosis, with a methodology based on decisions fusion. The main issue concerns the integration of different decisions emanating from individual monitoring methods in order to obtain more reliable results. The methodology is based on a theoretical learning of the Bayesian network parameters, according to monitoring objectives to be reached. The development leads to a multi-objective problem under constraints, which is solved with a lexicographic approach. The first step is offline and consists of defining the objectives to be achieved in order to improve the overall performance of the system. The Bayesian network parameters respecting these objectives are then deduced theoretically. Finally, the parametrized Bayesian network is used online to test the decision fusion performances. These performances are evaluated in terms of Fault Diagnostic Rate (FDR) and False Alarm Rate (FAR) for the detection and diagnosis stage, and in terms of Remaining Useful Life (RUL) for the prognosis.

Pedestrian detection is a canonical instance of object detection that remains a popular topic of research and a key problem in computer vision due to its diverse applications. These applications have the potential to positively improve the quality of life. In recent years, the number of approaches to detecting pedestrians in monocular and binocular images has grown steadily. However, the use of multispectral imaging is still uncommon. This thesis work presents a novel approach to data and feature fusion of a multispectral imaging system for pedestrian detection. It also includes the design and building of a test rig which allows for quick data collection of real-world driving. An application of the mathematical theory of trifocal tensor is used to post process this data. This allows for pixel level data fusion across a multispectral set of data. Performance results based on commonly used SVM classification architectures are evaluated against the collected data set. Lastly, a novel cascaded SVM architecture used in both classification and detection is discussed. Performance improvements through the use of feature fusion is demonstrated.

Prediction of protein secondary structure from primary sequence of amino acids is a very challenging task, and the problem has been approached from several angles. Proteins have many different biological functions; they may act as enzymes or as building blocks (muscle fibers) or may have transport function (e.g., transport of oxygen). The three-dimensional protein structure determines the functional properties of the protein. A lot of interesting work has been done on this problem, and over the last 10 to 20 years the methods have gradually improved in accuracy. In this dissertation we investigate several techniques for predicting the protein secondary structure. The prediction is carried out mainly using pattern classification techniques such as neural networks, genetic algorithms, simulated annealing. Each individual algorithm may work well in certain situations but fails in others. Capitalizing on the positive decisions can be achieved by forcing the various methods to collaborate to reach a unified consensus based on their previous performances. The process of combining classifiers is called decision fusion. The various decision fusion techniques such as the committee method, correlation method and the Bayesian inference methods to fuse the solutions from various approaches and to get better prediction accuracy are thoroughly explored in this dissertation. The RS126 data set was used for training and testing purposes. The results of applying pattern classification algorithms along with decision fusion techniques showed improvement in the prediction accuracy compared to that of prediction by neural networks or pattern classification algorithms individually or combined with neural networks. This research has shown that decision fusion techniques can be used to obtain better protein secondary structure prediction accuracy.

In multi-level decision making (DM) approaches, the final decision is reached by going through a finite number of DM levels. Usually, in each level, a raw decision is produced first and then a suitable decision fusion technique is employed to merge the lower level decisions with the raw decision in the construction of the final decision of the present level. The basic difficulty in these approaches is the determination of how the consecutive levels should interact with each other. In this thesis, two different multi-level DM models have been proposed. The main idea in the first model, &ldquo / hierarchical DM&rdquo / (HDM), is to transfer the decisions of previous hierarchical levels to an upper hierarchy with some reliability values. These decisions are then fused using a suitable decision fusion technique to attain more consistent decisions at an upper level. The second model &ldquo / local DM in multiplelevels&rdquo / (LDM-ML) depends on what may be called as local DM process. Instead of designing an agent to perform globally, designing relatively simple agents which are supposed to work in local regions is the essence of the second idea. Final decision is partially constructed by contribution of a sufficient number of local DM agents. A successful local agent is retained in the agent pool whereas a local agent not successful enough is eliminated and removed from the agent pool. These models have been applied on two case studies associated with fault detection in a four-tank system and prediction of lotto sales.

The energy efficiency issue, which is subjected to an energy constraint, is important for the applications in wireless sensor network. For the distributed detection problem considered in this thesis, the sensor makes a local decision based on its observation and transmits a one-bit message to the fusion center. We consider the local sensors employing a censoring scheme, where the sensors are silent and transmit nothing to fusion center if their observations are not very informative. The goal of this thesis is to achieve an energy efficiency design when the distributed detection employs the censoring scheme. Simulation results show that we can have the same error probabilities of decision fusion while conserving more energy simultaneously as compared with the detection without using censoring schemes. In this thesis, we also demonstrate that the error probability of decision fusion is a convex function of the censoring probability.

The ongoing development and increased affordability of hyperspectral sensors are increasing their utilization in a variety of applications, such as agricultural monitoring and decision making. Hyperspectral Automated Target Recognition (ATR) systems typically rely heavily on dimensionality reduction methods, and particularly intelligent reduction methods referred to as feature extraction techniques. This dissertation reports on the development, implementation, and testing of new hyperspectral analysis techniques for ATR systems, including their use in agricultural applications where ground truthed observations available for training the ATR system are typically very limited. This dissertation reports the design of effective methods for grouping and down-selecting Discrete Wavelet Transform (DWT) coefficients and the design of automated Wavelet Packet Decomposition (WPD) filter tree pruning methods for use within the framework of a Multiclassifiers and Decision Fusion (MCDF) ATR system. The efficacy of the DWT MCDF and WPD MCDF systems are compared to existing ATR methods commonly used in hyperspectral remote sensing applications. The newly developed methods’ sensitivity to operating conditions, such as mother wavelet selection, decomposition level, and quantity and quality of available training data are also investigated. The newly developed ATR systems are applied to the problem of hyperspectral remote sensing of agricultural food crop contaminations either by airborne chemical application, specifically Glufosinate herbicide at varying concentrations applied to corn crops, or by biological infestation, specifically soybean rust disease in soybean crops. The DWT MCDF and WPD MCDF methods significantly outperform conventional hyperspectral ATR methods. For example, when detecting and classifying varying levels of soybean rust infestation, stepwise linear discriminant analysis, results in accuracies of approximately 30%-40%, but WPD MCDF methods result in accuracies of approximately 70%-80%.

The technique of sensor fusion addresses the issues relating to the optimality of decision-making in the multiple-sensor framework. The advances in sensor fusion enable to perform intrusion detection for both rare and new attacks. This thesis discusses this assertion in detail, and describes the theoretical and experimental work done to show its validity. The attack-detector relationship is initially modeled and validated to understand the detection scenario. The different metrics available for the evaluation of intrusion detection systems are also introduced. The usefulness of the data set used for experimental evaluation has been demonstrated. The issues connected with intrusion detection systems are analyzed and the need for incorporating multiple detectors and their fusion is established in this work. Sensor fusion provides advantages with respect to reliability and completeness, in addition to intuitive and meaningful results. The goal for this work is to investigate how to combine data from diverse intrusion detection systems in order to improve the detection rate and reduce the false-alarm rate. The primary objective of the proposed thesis work is to develop a theoretical and practical basis for enhancing the performance of intrusion detection systems using advances in sensor fusion with easily available intrusion detection systems. This thesis introduces the mathematical basis for sensor fusion in order to provide enough support for the acceptability of sensor fusion in performance enhancement of intrusion detection systems. The thesis also shows the practical feasibility of performance enhancement using advances in sensor fusion and discusses various sensor fusion algorithms, its characteristics and related design and implementation is-sues. We show that it is possible to build performance enhancement to intrusion detection systems by setting proper threshold bounds and also by rule-based fusion. We introduce an architecture called the data-dependent decision fusion as a framework for building intrusion detection systems using sensor fusion based on data-dependency. Furthermore, we provide information about the types of data, the data skewness problems and the most effective algorithm in detecting different types of attacks. This thesis also proposes and incorporates a modified evidence theory for the fusion unit, which performs very well for the intrusion detection application. The future improvements in individual IDSs can also be easily incorporated in this technique in order to obtain better detection capabilities. Experimental evaluation shows that the proposed methods have the capability of detecting a significant percentage of rare and new attacks. The improved performance of the IDS using the algorithms that has been developed in this thesis, if deployed fully would contribute to an enormous reduction of the successful attacks over a period of time. This has been demonstrated in the thesis and is a right step towards making the cyber space safer.

<p>A series of fusion techniques are developed and applied to EEG and pupillary recording analysis in a rapid serial visual presentation (RSVP) based image triage task, in order to improve the accuracy of capturing single-trial neural/pupillary signatures (patterns) associated with visual target detection.</p><p>The brain response to visual stimuli is not a localized pulse, instead it reflects time-evolving neurophysiological activities distributed selectively in the brain. To capture the evolving spatio-temporal pattern, we divide an extended (``global") EEG data epoch, time-locked to each image stimulus onset, into multiple non-overlapping smaller (``local") temporal windows. While classifiers can be applied on EEG data located in multiple local temporal windows, outputs from local classifiers can be fused to enhance the overall detection performance.</p><p>According to the concept of induced/evoked brain rhythms, the EEG response can be decomposed into different oscillatory components and the frequency characteristics for these oscillatory components can be evaluated separately from the temporal characteristics. While the temporal-based analysis achieves fairly accurate detection performance, the frequency-based analysis can improve the overall detection accuracy and robustness further if frequency-based and temporal-based results are fused at the decision level.</p><p>Pupillary response provides another modality for a single-trial image triage task. We developed a pupillary response feature construction and selection procedure to extract/select the useful features that help to achieve the best classification performance. The classification results based on both modalities (pupillary and EEG) are further fused at the decision level. Here, the goal is to support increased classification confidence through inherent modality complementarities. The fusion results show significant improvement over classification results using any single modality.</p><p>For crucial image triage tasks, multiple image analysts could be asked to evaluate the same set of images to improve the probability of detection and reduce the probability of false positive. We observe significant performance gain by fusing the decisions drawn by multiple analysts.</p><p>To develop a practical real-time EEG-based application system, sometimes we have to work with an EEG system that has a limited number of electrodes. We present methods of ranking the channels, identifying a reduced set of EEG channels that can deliver robust classification performance.</p> / Dissertation