Event detection is a critical challenge in many fields like video surveillance, social graph analysis, and multimedia processing. Furthermore, events are “structured” objects involv ing multiple components like the event type, the participants with their roles, and the atomic events in which it decomposes. Therefore, the recognition of an event is not only limited to recognize the type of the event and when it happened, but it involves solving a set of simple tasks. Exploiting background knowledge about events and their relations could then be beneficial for event detection. In the last years, neuro-symbolic integration has been proposed to merge the strengths and overcome the drawbacks of both symbolic and neural worlds. As a consequence, different neuro-symbolic frameworks, which com bine low-level perception of neural networks with a symbolic layer, encoding prior domain knowledge (usually defined in terms of logical rules), have been applied to solve different atemporal tasks. In this thesis, we want to investigate the application of the neuro-symbolic paradigm for event detection. This would also provide a better insight into the strengths and limitations of neuro-symbolic towards tasks involving time.

Master of Science / Department of Computing and Information Sciences / William H. Hsu / Global epidemic surveillance is an essential task for national biosecurity management and bioterrorism prevention. The main goal is to protect the public from major health threads. To perform this task effectively one requires reliable, timely and accurate medical information from a wide range of sources. Towards this goal, we present a framework for epidemiological analytics that can be used to extract and visualize infectious disease outbreaks from the variety of unstructured web sources automatically. More precisely, in this thesis, we consider several research tasks including document relevance classification, entity extraction and animal disease-related event recognition in the veterinary epidemiology domain. First, we crawl web sources and classify collected documents by topical relevance using supervised learning algorithms. Next, we propose a novel approach for automated ontology construction in the veterinary medicine domain. Our approach is based on semantic relationship discovery using syntactic patterns. We then apply our automatically-constructed ontology for the domain-specific entity extraction task. Moreover, we compare our ontology-based entity extraction results with an alternative sequence labeling approach. We introduce a sequence labeling method for the entity tagging that relies on syntactic feature extraction using a sliding window. Finally, we present our novel sentence-based event recognition approach that includes three main steps: entity extraction of animal diseases, species, locations, dates and the confirmation status n-grams; event-related sentence classification into two categories - suspected or confirmed; automated event tuple generation and aggregation. We show that our document relevance classification results as well as entity extraction and disease-related event recognition results are significantly better compared to the results reported by other animal disease surveillance systems.

In this thesis, we present an event orderer application that works on Turkish texts. Events are words denoting an occurrence or happenings in natural language texts. By using the features of the events in a sentence or by the helps of temporal expressions in the sentence, anchoring an event on a timeline or ordering events between other events are called event ordering. The application presented in this thesis, is one of the earliest study in this domain with Turkish and it realizes all needed sub modules for event ordering. It realizes event recognition in Turkish texts and event feature detection in Turkish texts. In addition to this, the application is realizing temporal expression recognition and temporal signal recognition tasks.

Regularization techniques have become a principled tool for model-based statistics and artificial intelligence research. However, in most situations, these regularization terms are not well interpreted, especially on how they are related to the loss function and data matrix in a given statistic model. In this work, we propose a robust minimax formulation to interpret the relationship between data and regularization terms for a large class of loss functions. We show that various regularization terms are essentially corresponding to different distortions to the original data matrix. This supplies a unified framework for understanding various existing regularization terms, designing novel regularization terms based on perturbation analysis techniques, and inspiring novel generic algorithms. To show how to apply minimax related concepts to real-world learning tasks, we develop a new fault-tolerant classification framework to combat class noise for general multi-class classification problems; further, by studying the relationship between the majorizable function class and the minimax framework, we develop an accurate, efficient, and scalable algorithm for solving a large family of learning formulations. In addition, this work has been further extended to tackle several important matrix-decomposition-related learning tasks, and we have validated our work on various real-world applications including structure-from-motion (with missing data) and latent structure dictionary learning tasks. This work, composed of a unified formulation, a scalable algorithm, and promising applications in many real-world learning problems, contributes to the understanding of various hidden robustness in many learning models. As we show, many classical statistical machine learning models can be unified using this formulation and accurate, efficient, and scalable algorithms become available from our research.

This dissertation addresses two fundamental problems in computer vision—namely, multitarget tracking and event recognition in videos. These problems are challenging because uncertainty may arise from a host of sources, including motion blur, occlusions, and dynamic cluttered backgrounds. We show that these challenges can be successfully addressed by using a multiscale, volumetric video representation, and taking into account various constraints between events offered by domain knowledge. The dissertation presents our two alternative approaches to multitarget tracking. The first approach seeks to transitively link object detections across consecutive video frames by finding the maximum independent set of a graph of all object detections. Two maximum-independent-set algorithms are specified, and their convergence properties theoretically analyzed. The second approach hierarchically partitions the space-time volume of a video into tracks of objects, producing a segmentation graph of that video. The resulting tracks encode rich contextual cues between salient video parts in space and time, and thus facilitate event recognition, and segmentation in space and time. We also describe our two alternative approaches to event recognition. The first approach seeks to learn a structural probabilistic model of an event class from training videos represented by hierarchical segmentation graphs. The graph model is then used for inference of event occurrences in new videos. Learning and inference algorithms are formulated within the same framework, and their convergence rates theoretically analyzed. The second approach to event recognition uses probabilistic first-order logic for reasoning over continuous time intervals. We specify the syntax, learning, and inference algorithms of this probabilistic event logic. Qualitative and quantitative results on benchmark video datasets are also presented. The results demonstrate that our approaches provide consistent video interpretation with respect to acquired domain knowledge. We outperform most of the state-of-the-art approaches on benchmark datasets. We also present our new basketball dataset that complements existing benchmarks with new challenges. / Graduation date: 2011 / Access restricted to the OSU Community at author's request from May 12, 2011 - May 12, 2012

This thesis is devoted to the problems of defining and developing the basic building blocks of an automated surveillance system. As its initial step, a background-modeling algorithm is described for segmenting moving objects from the background, which is capable of adapting to dynamic scene conditions, as well as determining shadows of the moving objects. After obtaining binary silhouettes for targets, object association between consecutive frames is achieved by a hypothesis-based tracking method. Both of these tasks provide basic information for higher-level processing, such as activity analysis and object identification. In order to recognize the nature of an event occurring in a scene, hidden Markov models (HMM) are utilized. For this aim, object trajectories, which are obtained through a successful track, are written as a sequence of flow vectors that capture the details of instantaneous velocity and location information. HMMs are trained with sequences obtained from usual motion patterns and abnormality is detected by measuring the distance to these models. Finally, MPEG-7 visual descriptors are utilized in a regional manner for object identification. Color structure and homogeneous texture parameters of the independently moving objects are extracted and classifiers, such as Support Vector Machine (SVM) and Bayesian plug-in (Mahalanobis distance), are utilized to test the performance of the proposed person identification mechanism. The simulation results with all the above building blocks give promising results, indicating the possibility of constructing a fully automated surveillance system for the future.

L'explosion de la quantité de documents multimédias, suite à l'essor des technologies numériques, a rendu leur indexation très coûteuse et manuellement impossible. Par conséquent, le besoin de disposer de systèmes d'indexation capables d'analyser, de stocker et de retrouver les documents multimédias automatiquement, et en se basant sur leur contenu (audio, visuel), s'est fait ressentir dans de nombreux domaines applicatifs. Cependant, les techniques d'indexation actuelles rencontrent encore des problèmes de faisabilité ou de qualité. Leur performance reste très limitée et est dépendante de plusieurs facteurs comme la variabilité et la quantité de données à traiter. En effet, les systèmes d'indexation cherchent à reconnaître des concepts statiques, comme des objets (vélo, chaise,...), ou des événements (mariage, manifestation,...). Ces systèmes se heurtent donc au problème de variabilité de formes, de positions, de poses, d'illuminations, d'orientations des objets. Le passage à l'échelle pour pouvoir traiter de très grands volumes de données tout en respectant des contraintes de temps de calcul et de stockage est également une contrainte.Dans cette thèse, nous nous intéressons à l'amélioration de la performance globale de ces systèmes d'indexation de documents multimédias par le contenu. Pour cela nous abordons le problème sous différents angles et apportons quatre contributions à divers stades du processus d'indexation. Nous proposons tout d'abord une nouvelle méthode de fusion "doublement précoce " entre différentes modalités ou différentes sources d'informations afin d'exploiter au mieux la corrélation entre les modalités. Cette méthode est ensuite appliquée à la détection de scènes violentes dans les films. Nous développons ensuite une méthode faiblement supervisée pour la localisation des concepts basiques (comme les objets) dans les images qui pourra être utilisé plus tard comme un descripteur et une information supplémentaire pour la détection de concepts plus complexes (comme des événements). Nous traitons également la problématique de réduction du bruit généré par des annotations ambiguës sur les données d'apprentissage en proposant deux méthodes: une génération de nouvelles annotations au niveau des plans et une méthode de pondération des plans. Enfin, nous avons mis en place une méthode d'optimisation des représentations du contenu multimédia qui combine une réduction de dimension basée sur une ACP et des transformations non linéaires.Les quatre contributions sont testées et évaluées sur les collections de données faisant référence dans le domaine, comme TRECVid ou MediaEval. Elles ont participé au bon classement de nos soumissions dans ces campagnes. / A consequence of the rise of digital technology is that the quantity of available collections of multimedia documents is permanently and strongly increasing. The indexing of these documents became both very costly and impossible to do manually. In order to be able to analyze, classify and search multimedia documents, indexing systems have been defined. However, most of these systems suffer quality or practicability issues. Their performance is limited and depends on the data volume and data variability. Indexing systems analyze multimedia documents, looking for static concepts (bicycle, chair,...), or events (wedding, protest,...). Therefore, the variability in shapes, positions, lighting or orientation of objects hinders the process. Another aspect is that systems must be scalable. They should be able to handle big data while using reasonable amount of computing time and memory.The aim of this thesis is to improve the general performance of content-based multimedia indexing systems. Four main contributions are brought in this thesis for improving different stages of the indexing process. The first one is an "early-early fusion method" that merges different information sources in order to extract their deep correlations. This method is used for violent scenes detection in movies. The second contribution is a weakly supervised method for basic concept (objects) localization in images. This can be used afterwards as a new descriptor to help detecting complex concepts (events). The third contribution tackles the noise reduction problem on ambiguously annotated data. Two methods are proposed: a shot annotation generator, and a shot weighing method. The last contribution is a generic descriptor optimization method, based on PCA and non-linear transforms.These four contributions are tested and evaluated using reference data collections, including TRECVid and MediaEval. These contributions helped our submissions achieving very good rankings in those evaluation campaigns.

Master of Science / Department of Computing and Information Sciences / William H. Hsu / In addition to named entities such as persons, locations, organizations, and quantities which convey factual information, there are other entities and attributes that relate identifiable objects in the text and can provide valuable additional information. In the field of epizootics, these include specific properties of diseases such as their name, location, species affected, and current confirmation status. These are important for compiling the spatial and temporal statistics and other information needed to track diseases, leading to applications such as detection and prevention of bioterrorism. Toward this objective, we present a system (Rule Based Event Extraction System in Epizootic Domains) that can be used for extracting the infectious disease outbreaks from the unstructured data automatically by using the concept of pattern matching. In addition to extracting events, the components of this system can help provide structured and summarized data that can be used to differentiate confirmed events from suspected events, answer questions regarding when and where the disease was prevalent develop a model for predicting future disease outbreaks, and support visualization using interfaces such as Google Maps. While developing this system, we consider the research issues that include document relevance classification, entity extraction, recognizing the outbreak events in the disease domain and to support the visualization for events. We present a sentence-based event extraction approach for extracting the outbreak events from epizootic domain that has tasks such as extracting the events such as the disease name, location, species, confirmation status, and date; classifying the events into two categories of confirmation status- confirmed or suspected. The present approach shows how confirmation status is important in extracting the disease based events from unstructured data and a pyramid approach using reference summaries is used for evaluating the extracted events.

In this thesis, novel methods for background modeling, tracking, occlusion handling and event recognition via multi-camera configurations are presented. As the initial step, building blocks of typical single camera surveillance systems that are moving object detection, tracking and event recognition, are discussed and various widely accepted methods for these building blocks are tested to asses on their performance. Next, for the multi-camera surveillance systems, background modeling, occlusion handling, tracking and event recognition for two-camera configurations are examined. Various foreground detection methods are discussed and a background modeling algorithm, which is based on multi-variate mixture of Gaussians, is proposed. During occlusion handling studies, a novel method for segmenting the occluded objects is proposed, in which a top-view of the scene, free of occlusions, is generated from multi-view data. The experiments indicate that the occlusion handling algorithm operates successfully on various test data. A novel tracking method by using multi-camera configurations is also proposed. The main idea of multi-camera employment is fusing the 2D information coming from the cameras to obtain a 3D information for better occlusion handling and seamless tracking. The proposed algorithm is tested on different data sets and it shows clear improvement over single camera tracker. Finally, multi-camera trajectories of objects are classified by proposed multi-camera event recognition method. In this method, concatenated different view trajectories are used to train Gaussian Mixture Hidden Markov Models. The experimental results indicate an improvement for the multi-camera event recognition performance over the event recognition by using single camera.

Vidéo d'interprétation et de compréhension est l'un des objectifs de recherche à long terme dans la vision par ordinateur. Vidéos réalistes tels que les films présentent une variété de problèmes difficiles d'apprentissage machine, telles que la classification d'action / récupération d'action, de suivi humaines, la classification interaction homme / objet, etc Récemment robustes descripteurs visuels pour la classification vidéo ont été développés, et ont montré qu'il est possible d'apprendre classificateurs visuels réalistes des paramètres difficile. Toutefois, afin de déployer des systèmes de reconnaissance visuelle à grande échelle dans la pratique, il devient important d'aborder l'évolutivité des techniques. L'objectif principal est cette thèse est de développer des méthodes évolutives pour l'analyse de contenu vidéo (par exemple pour le classement ou la classification). / Video interpretation and understanding is one of the long-term research goals in computer vision. Realistic videos such as movies present a variety of challenging machine learning problems, such as action classification/action retrieval, human tracking, human/object interaction classification, etc. Recently robust visual descriptors for video classification have been developed, and have shown that it is possible to learn visual classifiers in realistic difficult settings. However, in order to deploy visual recognition systems on large-scale in practice it becomes important to address the scalability of the techniques. The main goal is this thesis is to develop scalable methods for video content analysis (eg for ranking, or classification).