A critical requirement for achieving ubiquity of artificial intelligence is to provide intelligent environments with the ability to recognize and understand human behaviour. If this is achieved, proactive interaction can occur and, more interestingly, a great variety of services can be developed. In this thesis we aim to support the development of ambient-assisted living services with advances in human behaviour analysis. Specifically, visual data analysis is considered in order to detect and understand human activity at home. As part of an intelligent monitoring system, single- and multi-view recognition of human actions is performed, along several optimizations and extensions. The present work may pave the way for more advanced human behaviour analysis techniques, such as the recognition of activities of daily living, personal routines and abnormal behaviour detection.

This thesis is concerned with the Distributed Information Gathering (DIG) problem in which a Sensor Network is tasked with building a common representation of environment. The problem is motivated by the advantages offered by distributed autonomous sensing systems and the challenges they present. The focus of this study is on Macro Sensor Networks, characterized by platform mobility, heterogeneous teams, and long mission duration. The system under consideration may consist of an arbitrary number of mobile autonomous robots, stationary sensor platforms, and human operators, all linked in a network. This work describes a comprehensive framework called Active Sensor Network (ASN) which addresses the tasks of information fusion, decistion making, system configuration, and user interaction. The main design objectives are scalability with the number of robotic platforms, maximum flexibility in implementation and deployment, and robustness to component and communication failure. The framework is described from three complementary points of view: architecture, algorithms, and implementation. The main contribution of this thesis is the development of the ASN architecture. Its design follows three guiding principles: decentralization, modularity, and locality of interactions. These principles are applied to all aspects of the architecture and the framework in general. To achieve flexibility, the design approach emphasizes interactions between components rather than the definition of the components themselves. The architecture specifies a small set of interfaces sufficient to implement a wide range of information gathering systems. In the area of algorithms, this thesis builds on the earlier work on Decentralized Data Fusion (DDF) and its extension to information-theoretic decistion making. It presents the Bayesian Decentralized Data Fusion (BDDF) algorithm formulated for environment features represented by a general probability density function. Several specific representations are also considered: Gaussian, discrete, and the Certainty Grid map. Well known algorithms for these representations are shown to implement various aspects of the Bayesian framework. As part of the ASN implementation, a practical indoor sensor network has been developed and tested. Two series of experiments were conducted, utilizing two types of environment representation: 1) point features with Gaussian position uncertainty and 2) Certainty Grid maps. The network was operational for several days at a time, with individual platforms coming on and off-line. On several occasions, the network consisted of 39 software components. The lessons learned during the system's development may be applicable to other heterogeneous distributed systems with data-intensive algorithms.

The recent emergence of intelligent agent technology and advances in information gathering have been the important steps forward in efficiently managing and using the vast amount of information now available on the Web to make informed decisions. There are, however, still many problems that need to be overcome in the information gathering research arena to enable the delivery of relevant information required by end users. Good decisions cannot be made without sufficient, timely, and correct information. Traditionally it is said that knowledge is power, however, nowadays sufficient, timely, and correct information is power. So gathering relevant information to meet user information needs is the crucial step for making good decisions. The ideal goal of information gathering is to obtain only the information that users need (no more and no less). However, the volume of information available, diversity formats of information, uncertainties of information, and distributed locations of information (e.g. World Wide Web) hinder the process of gathering the right information to meet the user needs. Specifically, two fundamental issues in regard to efficiency of information gathering are mismatch and overload. The mismatch means some information that meets user needs has not been gathered (or missed out), whereas, the overload means some gathered information is not what users need. Traditional information retrieval has been developed well in the past twenty years. The introduction of the Web has changed people's perceptions of information retrieval. Usually, the task of information retrieval is considered to have the function of leading the user to those documents that are relevant to his/her information needs. The similar function in information retrieval is to filter out the irrelevant documents (or called information filtering). Research into traditional information retrieval has provided many retrieval models and techniques to represent documents and queries. Nowadays, information is becoming highly distributed, and increasingly difficult to gather. On the other hand, people have found a lot of uncertainties that are contained in the user information needs. These motivate the need for research in agent-based information gathering. Agent-based information systems arise at this moment. In these kinds of systems, intelligent agents will get commitments from their users and act on the users behalf to gather the required information. They can easily retrieve the relevant information from highly distributed uncertain environments because of their merits of intelligent, autonomy and distribution. The current research for agent-based information gathering systems is divided into single agent gathering systems, and multi-agent gathering systems. In both research areas, there are still open problems to be solved so that agent-based information gathering systems can retrieve the uncertain information more effectively from the highly distributed environments. The aim of this thesis is to research the theoretical framework for intelligent agents to gather information from the Web. This research integrates the areas of information retrieval and intelligent agents. The specific research areas in this thesis are the development of an information filtering model for single agent systems, and the development of a dynamic belief model for information fusion for multi-agent systems. The research results are also supported by the construction of real information gathering agents (e.g., Job Agent) for the Internet to help users to gather useful information stored in Web sites. In such a framework, information gathering agents have abilities to describe (or learn) the user information needs, and act like users to retrieve, filter, and/or fuse the information. A rough set based information filtering model is developed to address the problem of overload. The new approach allows users to describe their information needs on user concept spaces rather than on document spaces, and it views a user information need as a rough set over the document space. The rough set decision theory is used to classify new documents into three regions: positive region, boundary region, and negative region. Two experiments are presented to verify this model, and it shows that the rough set based model provides an efficient approach to the overload problem. In this research, a dynamic belief model for information fusion in multi-agent environments is also developed. This model has a polynomial time complexity, and it has been proven that the fusion results are belief (mass) functions. By using this model, a collection fusion algorithm for information gathering agents is presented. The difficult problem for this research is the case where collections may be used by more than one agent. This algorithm, however, uses the technique of cooperation between agents, and provides a solution for this difficult problem in distributed information retrieval systems. This thesis presents the solutions to the theoretical problems in agent-based information gathering systems, including information filtering models, agent belief modeling, and collection fusions. It also presents solutions to some of the technical problems in agent-based information systems, such as document classification, the architecture for agent-based information gathering systems, and the decision in multiple agent environments. Such kinds of information gathering agents will gather relevant information from highly distributed uncertain environments.

Advances in information technology produce large sets of data for decision makers. In both military and civilian efforts to achieve decision superiority, decision makers have to act agilely with proper, adequate and relevant information available. Information fusion is a process aimed to support decision makers’ situation awareness. This involves a process of combining data and information from disparate sources with prior information or knowledge to obtain an improved state estimate about an agent or other relevant phenomena. The important issue in decision making is not only assessing the current situation but also envisioning how a situation may evolve. In this work we focus on the prediction part of decision making called predictive situation awareness. We introduce new methodology where simulations and plan recognition are tools for achieving improved predictive situation awareness. Plan recognition is the term given to the process of inferring an agent’s intentions from a set of actions and is intended to support decision making. Beside its main task that is to support decision makers’ predictive situation awareness, plan recognition could also be used for coordination of actions and for developing computer-game agents that possess cognitive ability to recognize other agents’ behaviour. Successful plan recognition is heavily dependent on the data that is supplied. Therefore we introduce a bridge between plan recognition and sensor management where results of our plan recognition are reused to the control of, to give focus of attention to, the sensors that are expected to acquire the most important/relevant information. Our methodologies include knowledge representation, embedded stochastic simulations, microeconomics, imprecise knowledge and statistical inference issues. / QC 20100922

In recent years, due to the increasing usage of videos, manual information extraction is becoming insufficient to users. Therefore, extracting semantic information automatically turns out to be a serious requirement. Today, there exists some systems that extract semantic information automatically by using visual, auditory and textual data separately but the number of studies that uses more than one data source is very limited. As some studies on this topic have already shown, using multimodal video data for automatic information extraction ensures getting better results by guaranteeing increase in the accuracy of semantic information that is retrieved from visual, auditory and textual sources. In this thesis, a complete system which fuses the semantic information that is obtained from visual, auditory and textual video data is introduced. The fusion system carries out the following procedures / analyzing and uniting the semantic information that is extracted from multimodal data by utilizing concept interactions and consequently generating a semantic dataset which is ready to be stored in a database. Besides, experiments are conducted to compare results obtained from the proposed multimodal fusion operation with results obtained as an outcome of semantic information extraction from just one modality and other fusion methods. The results indicate that fusing all available information along with concept relations yields better results than any unimodal approaches and other traditional fusion methods in overall.

<p>To incorporate new technical advances into military domain and make those processes more <i>efficient</i> in accuracy, time and cost, a new concept of Network Centric Warfare has been introduced in the US military forces. In Sweden a similar concept has been studied under the name Network Based Defence (NBD). Here we present one of the methodologies, called tactical plan recognition that is aimed to support NBD in future.</p><p>Advances in sensor technology and modelling produce large sets of data for decision makers. To achieve <i>decision superiority</i>, decision makers have to act agile with proper, adequate and relevant information (data aggregates) available. Information fusion is a process aimed to support decision makers’ situation awareness. This involves a process of combining data and information from disparate sources with <i>prior</i> information or knowledge to obtain an improved state estimate about an agent or phenomena. <i>Plan recognition</i> is the term given to the process of inferring an agent’s intentions from a set of actions and is intended to support decision making.</p><p>The aim of this work has been to introduce a methodology where prior (empirical) knowledge (e.g. behaviour, environment and organization) is represented and combined with sensor data to recognize plans/behaviours of an agent or group of agents. We call this methodology <i>multi-agent plan recognition</i>. It includes knowledge representation as well as imprecise and statistical inference issues.</p><p>Successful plan recognition in large scale systems is heavily dependent on the data that is supplied. Therefore we introduce a <i>bridge</i> between the plan recognition and sensor management where results of our plan recognition are reused to the control of, give <i>focus of attention</i> to, the sensors that are supposed to acquire most important/<i>relevant</i> information.</p><p>Here we combine different theoretical methods (Bayesian Networks, Unified Modeling Language and Plan Recognition) and apply them for tactical military situations for ground forces. The results achieved from several proof-ofconcept models show that it is possible to model and recognize behaviour of tank units.</p>

This thesis is part of the knowledge representation domain and modeling of uncertainty in a context of information fusion. The main idea is to use semantic tools and more specifically ontologies, not only to represent the general domain knowledge and observations, but also to represent the uncertainty that sources may introduce in their own observations. We propose to represent these uncertainties and semantic imprecision trough a metaontology (called DS-Ontology) based on the theory of belief functions. The contribution of this work focuses first on the definition of semantic inclusion and intersection operators for ontologies and on which relies the implementation of the theory of belief functions, and secondly on the development of a tool called FusionLab for merging semantic information within ontologies from the previous theorical development. These works have been applied within a European maritime surveillance project.

This thesis is concerned with the Distributed Information Gathering (DIG) problem in which a Sensor Network is tasked with building a common representation of environment. The problem is motivated by the advantages offered by distributed autonomous sensing systems and the challenges they present. The focus of this study is on Macro Sensor Networks, characterized by platform mobility, heterogeneous teams, and long mission duration. The system under consideration may consist of an arbitrary number of mobile autonomous robots, stationary sensor platforms, and human operators, all linked in a network. This work describes a comprehensive framework called Active Sensor Network (ASN) which addresses the tasks of information fusion, decistion making, system configuration, and user interaction. The main design objectives are scalability with the number of robotic platforms, maximum flexibility in implementation and deployment, and robustness to component and communication failure. The framework is described from three complementary points of view: architecture, algorithms, and implementation. The main contribution of this thesis is the development of the ASN architecture. Its design follows three guiding principles: decentralization, modularity, and locality of interactions. These principles are applied to all aspects of the architecture and the framework in general. To achieve flexibility, the design approach emphasizes interactions between components rather than the definition of the components themselves. The architecture specifies a small set of interfaces sufficient to implement a wide range of information gathering systems. In the area of algorithms, this thesis builds on the earlier work on Decentralized Data Fusion (DDF) and its extension to information-theoretic decistion making. It presents the Bayesian Decentralized Data Fusion (BDDF) algorithm formulated for environment features represented by a general probability density function. Several specific representations are also considered: Gaussian, discrete, and the Certainty Grid map. Well known algorithms for these representations are shown to implement various aspects of the Bayesian framework. As part of the ASN implementation, a practical indoor sensor network has been developed and tested. Two series of experiments were conducted, utilizing two types of environment representation: 1) point features with Gaussian position uncertainty and 2) Certainty Grid maps. The network was operational for several days at a time, with individual platforms coming on and off-line. On several occasions, the network consisted of 39 software components. The lessons learned during the system's development may be applicable to other heterogeneous distributed systems with data-intensive algorithms.

<p>The maritime industry is experiencing one of its longest and fastest periods of growth. Hence, the global maritime surveillance capacity is in a great need of growth as well. The detection of vessel activity is an important objective of the civil security domain. Detecting vessel activity may become problematic if audit data is uncertain. This thesis aims to investigate if Bayesian networks acquired from expert knowledge can detect activities with a signature-based detection approach. For this, a maritime pilot-boat scenario has been identified with a domain expert. Each of the scenario’s activities has been divided up into signatures where each signature relates to a specific Bayesian network information node. The signatures were implemented to find evidences for the Bayesian network information nodes. AIS-data with real world observations have been used for testing, which have shown that it is possible to detect the maritime pilot-boat scenario based on the taken approach.</p>

Dans cette thèse nous montrons comment améliorer la perception d’un robot humanoïde NAO en utilisant la fusion multi-capteurs. Nous avons proposé deux scénarios: la détection de la couleur et la reconnaissance d’objets colorés. Dans ces deux situations, nous utilisons la caméra du robot et nous ajoutons des caméras externes pour augmenter la fiabilité de la détection car nous nous plaçons dans un contexte expérimental dans lequel l’environnement est non contrôlé. Pour la détection de la couleur, l’utilisateur demande au robot NAO de trouver un objet coloré. La couleur est décrite par des termes linguistiques tels que: rouge, jaune, .... Le principal problème à résoudre est la façon dont le robot reconnaît les couleurs. Pour ce faire, nous avons proposé un système Flou de Sugeno pour déterminer la couleur demandée. Pour simplifier, les cibles choisies sont des balles colorées. Nous avons appliqué la transformation de Hough pour extraire les valeurs moyennes des pixels des balles détectées. Ces valeurs sont utilisées comme entrées pour le système Flou. Les fonctions d'appartenance et les règles d'inférence du système sont construites sur la base de l'évaluation perceptive de l'humain. La sortie du système Flou est une valeur numérique indiquant le nom de la couleur. Une valeur de seuil est introduite pour définir la zone de décision pour chaque couleur. Si la sortie floue tombe dans cet intervalle, alors la couleur est considérée comme la vraie sortie du système. Nous sommes dans un environnement non contrôlé dans lequel il y a des incertitudes et des imprécisions (variation de la lumière, qualité des capteurs, similarité entre couleurs). Ces facteurs affectent la détection de la couleur par le robot. L’introduction du seuil qui encadre la couleur, conduit à un compromis entre l'incertitude et la fiabilité. Si cette valeur est faible, les décisions sont plus fiables, mais le nombre de cas incertains augmente, et vice et versa. Dans nos expérimentations, on a pris une valeur de seuil petite, de sorte que l'incertitude soit plus importante, et donc la prise de décision par un capteur unique, celui de NAO, soit faible. Nous proposons d'ajouter d’autres caméras 2D dans le système afin d’améliorer la prise de décision par le robot NAO. Cette prise de décision résulte de la fusion des sorties des caméras en utilisant la théorie des fonctions de croyance pour lever les ambiguïtés. La valeur de seuil est prise en compte lors de la construction des valeurs de masse à partir de la sortie Floue de Sugeno de chaque caméra. La règle de combinaison de Dempster-Shafer et le maximum de probabilité pignistique sont choisis dans la méthode. Selon nos expériences, le taux de détection du système de fusion est grandement amélioré par rapport au taux de détection de chaque caméra prise individuellement. Nous avons étendu cette méthode à la reconnaissance d’objets colorés en utilisant des caméras hétérogènes 2D et 3D. Pour chaque caméra, nous extrayons vecteurs de caractéristiques (descripteurs SURF et SHOT) des objets, riches en informations caractérisant les modèles d'objets. Sur la base de la correspondance avec des modèles formés et stockés dans la base d'apprentissage, chaque vecteur de caractéristiques de l'objet détecté vote pour une ou plusieurs classes appartenant à l'ensemble de puissance. Nous construisons une fonction de masse après une étape de normalisation. Dans cette expérimentation, la règle de combinaison de Dempster-Shafer et le maximum de probabilité pignistique sont utilisés pour prendre la décision finale. A la suite des trois expérimentations réalisées, le taux de reconnaissance du système de fusion est bien meilleur que le taux de décision issu de chaque caméra individuellement. Nous montrons ainsi que la fusion multi-capteurs permet d’améliorer la prise de décision du robot. / Being interested in the important role of robotics in human life, we do a research about the improvement in reliability of a humanoid robot NAO by using multi-sensor fusion. In this research, we propose two scenarios: the color detection and the object recognition. In these two cases, a camera of the robot is used in combination with external cameras to increase the reliability under non-ideal working conditions. For the color detection, the NAO robot is requested to find an object whose color is described in human terms such as: red, yellow, brown, etc. The main problem to be solved is how the robot recognizes the colors as well as the human perception does. To do that, we propose a Fuzzy Sugeno system to decide the color of a detected target. For simplicity, the chosen targets are colored balls, so that the Hough transformation is employed to extract the average pixel values of the detected ball, then these values are used as the inputs for the Fuzzy system. The membership functions and inference rules of the system are constructed based on perceptual evaluation of human. The output of the Fuzzy system is a numerical value indicating a color name. Additionally, a threshold value is introduced to define the zone of decision for each color. If the Fuzzy output falls into a color interval constructed by the threshold value, that color is considered to be the output of the system. This is considered to be a good solution in an ideal condition, but not in an environment with uncertainties and imprecisions such as light variation, or sensor quality, or even the similarity among colors. These factors really affect the detection of the robot. Moreover, the introduction of the threshold value also leads to a compromise between uncertainty and reliability. If this value is small, the decisions are more reliable, but the number of uncertain cases are increases, and vice versa. However, the threshold value is preferred to be small after an experimental validation, so the need for a solution of uncertainty becomes more important. To do that, we propose adding more 2D cameras into the detection system of the NAO robot. Each camera applies the same method as described above, but their decisions are fused by using the Dempster-Shafer theory in order to improve the detection rate. The threshold value is taken into account to construct mass values from the Sugeno Fuzzy output of each camera. The Dempster-Shafer's rule of combination and the maximum of pignistic probability are chosen in the method. According to our experimens, the detection rate of the fusion system is really better than the result of each individual camera. We extend this recognition process for colored object recognition. These objects are previously learned during the training phase. To challenge uncertainties and imprecisions, the chosen objects look similar in many points: geometrical form, surface, color, etc. In this scenario, the recognition system has two 2D cameras: one of NAO and one is an IP camera, then we add a 3D camera to take the advantages of depth information. For each camera, we extract feature points of the objects (SURF descriptor for 2D data, and the SHOT descriptor for 3D data). To combine the cameras in the recognition system, the Dempster-Shafer theory is again employed for the fusion. Based on the correspondence to trained models stored in the learning base, each feature point of the detected object votes for one or several classes i.e. a hypothesis in the power set. We construct a mass function after a normalization step. In this case, the Dempster-Shafer's rule of combination and the maximum of pignistic probability are employed to make the final decision. After doing three experiments, we conclude that the recognition rate of the fusion system is much better than the rate of each individual camera, from that we confirm the benefits of multi-sensor fusion for the robot's reliability.