Peter Drucker, a well-known influential writer and philosopher in the field of management theory and practice, once claimed that “the most important thing in communication is hearing what isn't said.” It is not difficult to see that a similar concept also holds in the context of video scene understanding. In almost every non-trivial video scene, most important elements, such as the motives and intentions of the actors, can never be seen or directly observed, yet the identification of these latent attributes is crucial to our full understanding of the scene. That is to say, latent attributes matter. In this work, we explore the task of identifying latent attributes in video scenes, focusing on the mental states of participant actors. We propose a novel approach to the problem based on the use of large text collections as background knowledge and minimal information about the videos, such as activity and actor types, as query context. We formalize the task and a measure of merit that accounts for the semantic relatedness of mental state terms, as well as their distribution weights. We develop and test several largely unsupervised information extraction models that identify the mental state labels of human participants in video scenes given some contextual information about the scenes. We show that these models produce complementary information and their combination significantly outperforms the individual models, and improves performance over several baseline methods on two different datasets. We present an extensive analysis of our models and close with a discussion of our findings, along with a roadmap for future research.

Broadly this thesis looks into network and time-series mining problems pertaining to dynamics over networks in various domains. Which locations and staff should we monitor in order to detect C. Difficile outbreaks in hospitals? How do we predict the peak intensity of the influenza incidence in an interpretable fashion? How do we infer the states of all nodes in a critical infrastructure network where failures have occurred? Leveraging domain-based information should make it is possible to answer these questions. However, several new challenges arise, such as (a) presence of more complex dynamics. The dynamics over networks that we consider are complex. For example, C. Difficile spreads via both people-to-people and surface-to-people interactions and correlations between failures in critical infrastructures go beyond the network structure and depend on the geography as well. Traditional approaches either rely on models like Susceptible Infectious (SI) and Independent Cascade (IC) which are too restrictive because they focus only on single pathways or do not incorporate the model at all, resulting in sub-optimality. (b) data sparsity. Additionally, the data sparsity still persists in this space. Specifically, it is difficult to collect the exact state of each node in the network as it is high-dimensional and difficult to directly sample from. (c) mismatch between data and process. In many situations, the underlying dynamical process is unknown or depends on a mixture of several models. In such cases, there is a mismatch between the data collected and the model representing the dynamics. For example, the weighted influenza like illness (wILI) count released by the CDC, which is meant to represent the raw fraction of total population infected by influenza, actually depends on multiple factors like the number of health-care providers reporting the number and public tendency to seek medical advice. In such cases, methods which generalize well to unobserved (or unknown) models are required. Current approaches often fail in tackling these challenges as they either rely on restrictive models, require large volume of data, and/or work only for predefined models. In this thesis, we propose to leverage domain-based frameworks, which include novel models and analysis techniques, and domain-based low dimensional representation learning to tackle the challenges mentioned above for networks and time-series mining tasks. By developing novel frameworks, we can capture the complex dynamics accurately and analyze them more efficiently. For example, to detect C. Difficile outbreaks in a hospital setting, we use a two-mode disease model to capture multiple pathways of outbreaks and discrete lattice-based optimization framework. Similarly, we propose an information theoretic framework which includes geographically correlated failures in critical infrastructure networks to infer the status of the network components. Moreover, as we use more realistic frameworks to accurately capture and analyze the mechanistic processes themselves, our approaches are effective even with sparse data. At the same time, learning low-dimensional domain-aware embeddings capture domain specific properties (like incidence-based similarity between historical influenza seasons) more efficiently from sparse data, which is useful for subsequent tasks. Similarly, since the domain-aware embeddings capture the model information directly from the data without any modeling assumptions, they generalize better to new models. Our domain-aware frameworks and embeddings enable many applications in critical domains. For example, our domain-aware frameworks for C. Difficile allows different monitoring rates for people and locations, thus detecting more than 95% of outbreaks. Similarly, our framework for product recommendation in e-commerce for queries with sparse engagement data resulted in a 34% improvement over the current Walmart.com search engine. Similarly, our novel framework leads to a near optimal algorithms, with additive approximation guarantee, for inferring network states given a partial observation of the failures in networks. Additionally, by exploiting domain-aware embeddings, we outperform non-trivial competitors by up to 40% for influenza forecasting. Similarly, domain-aware representations of subgraphs helped us outperform non-trivial baselines by up to 68% in the graph classification task. We believe our techniques will be useful for variety of other applications in many areas like social networks, urban computing, and so on. / Doctor of Philosophy / Which locations and staff should we monitor to detect pathogen outbreaks in hospitals? How do we predict the peak intensity of the influenza incidence? How do we infer the failures in water distribution networks? These are some of the questions on dynamics over networks discussed in this thesis. Here, we leverage the domain knowledge to answer these questions. Specifically, we propose (a) novel optimization frameworks where we exploit domain knowledge for tractable formulations and near-optimal algorithms, and (b) low dimensional representation learning where we design novel neural architectures inspired by domain knowledge. Our frameworks capture the complex dynamics accurately and help analyze them more efficiently. At the same time, our low-dimensional embeddings capture domain specific properties more efficiently from sparse data, which is useful for subsequent tasks. Similarly, our domain-aware embeddings are inferred directly from the data without any modeling assumptions, hence they generalize better. The frameworks and embeddings we develop enable many applications in several domains. For example, our domain-aware framework for outbreak detection in hospitals has more than 95% accuracy. Similarly, our framework for product recommendation in e-commerce for queries with sparse data resulted in a 34% improvement over state-of-the-art e-commerce search engine. Additionally, our approach outperforms non-trivial competitors by up to 40% in influenza forecasting.

Gait analysis can improve our understanding of gait to improve medical diagnosis or treatment in clinical assessment. Studying the gait cycle in an embedded sensor system is essential for the detection of any abnormal walking pattern. This project aims to investigate several methods for gait phase recognition on embedded systems based on Hidden Markov Model (HMM) and Long short term memory (LSTM). This project proposes three methods, single HMM, multiple HMMs, and LSTM models, to identify the phase number in one gait. Single HMM has been constructed with the unit of gait via HMM learning. The corresponding phase number in the hidden state sequence can be selected for the observations via HMM decoding. Multiple HMMs have been constructed with the unit of phase instead of gait via HMM learning. The HMM evaluation can select the corresponding phase number in the hidden state sequence with the largest log- likelihood. Frame blocking and windowing function is also applied to evaluate these two methods. Estimation, validation, and forecast are implemented in the LSTM method as a benchmark. After comparing and evaluating the three methods for phase inference in terms of execution time, accuracy, and limitations, the method with multiple HMMs can provide satisfactory accuracy of gait phase number recognition in a relatively short time. It can be concluded that the multiple HMMs method may be more suitable for application in this phase inference scenario on the embedded sensor processing systems if the timing requirement is not so stringent. / Gånganalys kan förbättra vår förståelse för gång för att förbättra medicinsk diagnos eller behandling vid klinisk bedömning. Att studera gångcykeln i ett inbyggt sensorsystem är avgörande för detektering av onormalt gångmönster. Detta projekt syftar till att undersöka flera metoder för gångfasinferens på inbäddade system baserat på Hidden Markov Model (HMM) och Long short term memory (LSTM). I detta projekt har tre metoder, enstaka HMM, flera HMM och LSTM-modeller, föreslagits för att identifiera fasnumret i en gång. Enstaka HMM har konstruerats med gångenheten via HMM-lärande. Motsvarande fasnummer i den dolda tillståndssekvensen kan väljas för observationerna via HMM-avkodning. Flera HMM har konstruerats med fasenheten istället för gång via HMM-lärande. Motsvarande fasnummer i den dolda tillståndssekvensen kan väljas med störst logsannolikhet via HMM-utvärdering. Frame Blocking och Windowing-funktionen används också för att utvärdera dessa två metoder. Uppskattning, validering och prognos implementeras i LSTM-metoden som ett riktmärke. Efter att ha jämfört och utvärderat de tre metoderna för fasinferens när det gäller exekveringstid, noggrannhet och begränsningar kan metoden med flera HMM: er uppnå tillfredsställande noggrannhet för fasnummerigenkänning på relativt kort tid. Vi kan dra slutsatsen att den flera HMM-metoden kan vara mer lämplig för tillämpning i detta fasinferensscenario på de inbyggda sensorbehandlingssystemen om tidskravet inte är så strikt.

No description available.