The ever-increasing technological advances in embedded systems engineering, together with the proliferation of small-size sensor design and deployment, have enabled mobile devices (e.g., smartphones) to recognize daily occurring human based actions, activities and interactions. Therefore, inferring a vast variety of mobile device user based activities from a very diverse context obtained by a series of sensory observations has drawn much interest in the research area of ubiquitous sensing. The existence and awareness of the context provides the capability of being conscious of physical environments or situations around mobile device users, and this allows network services to respond proactively and intelligently based on such awareness. Hence, with the evolution of smartphones, software developers are empowered to create context aware applications for recognizing human-centric or community based innovative social and cognitive activities in any situation and from anywhere. This leads to the exciting vision of forming a society of ``Internet of Things" which facilitates applications to encourage users to collect, analyze and share local sensory knowledge in the purpose for a large scale community use by creating a smart network which is capable of making autonomous logical decisions to actuate environmental objects. More significantly, it is believed that introducing the intelligence and situational awareness into recognition process of human-centric event patterns could give a better understanding of human behaviors, and it also could give a chance for proactively assisting individuals in order to enhance the quality of lives. Mobile devices supporting emerging computationally pervasive applications will constitute a significant part of future mobile technologies by providing highly proactive services requiring continuous monitoring of user related contexts. However, the middleware services provided in mobile devices have limited resources in terms of power, memory and bandwidth as compared to the capabilities of PCs and servers. Above all, power concerns are major restrictions standing up to implementation of context-aware applications. These requirements unfortunately shorten device battery lifetimes due to high energy consumption caused by both sensor and processor operations. Specifically, continuously capturing user context through sensors imposes heavy workloads in hardware and computations, and hence drains the battery power rapidly. Therefore, mobile device batteries do not last a long time while operating sensor(s) constantly. In addition to that, the growing deployment of sensor technologies in mobile devices and innumerable software applications utilizing sensors have led to the creation of a layered system architecture (i.e., context aware middleware) so that the desired architecture can not only offer a wide range of user-specific services, but also respond effectively towards diversity in sensor utilization, large sensory data acquisitions, ever-increasing application requirements, pervasive context processing software libraries, mobile device based constraints and so on. Due to the ubiquity of these computing devices in a dynamic environment where the sensor network topologies actively change, it yields applications to behave opportunistically and adaptively without a priori assumptions in response to the availability of diverse resources in the physical world as well as in response to scalability, modularity, extensibility and interoperability among heterogeneous physical hardware. In this sense, this dissertation aims at proposing novel solutions to enhance the existing tradeoffs in mobile sensing between accuracy and power consumption while context is being inferred under the intrinsic constraints of mobile devices and around the emerging concepts in context-aware middleware framework.

Dissertation--Ruhr-Universität, Bochum, 2004. / Includes bibliographical references (p. 179-187).

Orientador: Alexandre César Rodrigues da Silva / Resumo: A utilização de sensores em ambientes inteligentes é fundamental para supervisionar as atividades dos seres humanos. No reconhecimento de atividades humanas, ou HAR (Human Activity Recognition), técnicas de supervisionamento são aplicadas para identificar as atividades realizadas em diversas aplicações, como no esporte e no acompanhamento de pessoas com necessidades especiais. O Sistema de Reconhecimento de Atividades Humanas (SIRAH) é empregado no reconhecimento de atividades humanas, utilizando um acelerômetro localizado na cintura da pessoa monitorada e uma Rede Neural Artificial para classificar sete atividades: em pé, deitado, sentado, caminhar, correr, sentar e levantar. Originalmente implementado no software MATLAB, realizava classificações offline em que os resultados não eram obtidos durante a execução das atividades. Apresenta-se, neste trabalho, o desenvolvimento de duas versões embarcadas do SIRAH, que executam o algoritmo de classificação durante a prática das atividades monitoradas. A primeira implementação foi efetuada no processador Nios II da Altera, que ofereceu a mesma exatidão do sistema offline com processamento limitado, pois o software consome 673 milissegundos para executar a classificação desejada. Para aprimorar o desempenho, outra versão foi implementada em FPGA utilizando a linguagem de descrição de hardware VHDL. O algoritmo de classificação opera em tempo real e é executado em apenas 236 microssegundos, garantindo total amostragem das acelerações... (Resumo completo, clicar acesso eletrônico abaixo) / Mestre

Dans le but d’interagir avec des êtres humains, les robots doivent effectuer destâches de perception basique telles que la détection de visage, l’estimation dela pose des personnes ou la reconnaissance de la parole. Cependant, pour interagir naturellement, avec les hommes, le robot doit modéliser des conceptsde haut niveau tels que les tours de paroles dans un dialogue, le centre d’intérêtd’une conversion, ou les interactions entre les participants. Dans ce manuscrit,nous suivons une approche ascendante (dite “top-down”). D’une part, nousprésentons deux méthodes de haut niveau qui modélisent les comportementscollectifs. Ainsi, nous proposons un modèle capable de reconnatre les activitésqui sont effectuées par différents des groupes de personnes conjointement, telsque faire la queue, discuter. Notre approche gère le cas général où plusieursactivités peuvent se dérouler simultanément et en séquence. D’autre part,nous introduisons une nouvelle approche d’apprentissage par renforcement deréseau de neurones pour le contrôle de la direction du regard du robot. Notreapproche permet à un robot d’apprendre et d’adapter sa stratégie de contrôledu regard dans le contexte de l’interaction homme-robot. Le robot est ainsicapable d’apprendre à concentrer son attention sur des groupes de personnesen utilisant seulement ses propres expériences (sans supervision extérieur).Dans un deuxième temps, nous étudions en détail les approchesd’apprentissage profond pour les problèmes de régression. Les problèmesde régression sont cruciaux dans le contexte de l’interaction homme-robotafin d’obtenir des informations fiables sur les poses de la tête et du corpsdes personnes faisant face au robot. Par conséquent, ces contributions sontvraiment générales et peuvent être appliquées dans de nombreux contextesdifférents. Dans un premier temps, nous proposons de coupler un mélangegaussien de régressions inverses linéaires avec un réseau de neurones convolutionnels. Deuxièmement, nous introduisons un modèle de mélange gaussien-uniforme afin de rendre l’algorithme d’apprentissage plus robuste aux annotations bruitées. Enfin, nous effectuons une étude à grande échelle pour mesurerl’impact de plusieurs choix d’architecture et extraire des recommandationspratiques lors de l’utilisation d’approches d’apprentissage profond dans destâches de régression. Pour chacune de ces contributions, une intense validation expérimentale a été effectuée avec des expériences en temps réel sur lerobot NAO ou sur de larges et divers ensembles de données. / In order to interact with humans, robots need to perform basic perception taskssuch as face detection, human pose estimation or speech recognition. However, in order have a natural interaction with humans, the robot needs to modelhigh level concepts such as speech turns, focus of attention or interactions between participants in a conversation. In this manuscript, we follow a top-downapproach. On the one hand, we present two high-level methods that model collective human behaviors. We propose a model able to recognize activities thatare performed by different groups of people jointly, such as queueing, talking.Our approach handles the general case where several group activities can occur simultaneously and in sequence. On the other hand, we introduce a novelneural network-based reinforcement learning approach for robot gaze control.Our approach enables a robot to learn and adapt its gaze control strategy inthe context of human-robot interaction. The robot is able to learn to focus itsattention on groups of people from its own audio-visual experiences.Second, we study in detail deep learning approaches for regression prob-lems. Regression problems are crucial in the context of human-robot interaction in order to obtain reliable information about head and body poses or theage of the persons facing the robot. Consequently, these contributions are really general and can be applied in many different contexts. First, we proposeto couple a Gaussian mixture of linear inverse regressions with a convolutionalneural network. Second, we introduce a Gaussian-uniform mixture model inorder to make the training algorithm more robust to noisy annotations. Finally,we perform a large-scale study to measure the impact of several architecturechoices and extract practical recommendations when using deep learning approaches in regression tasks. For each of these contributions, a strong experimental validation has been performed with real-time experiments on the NAOrobot or on large and diverse data-sets.

abstract: The data explosion in the past decade is in part due to the widespread use of rich sensors that measure various physical phenomenon -- gyroscopes that measure orientation in phones and fitness devices, the Microsoft Kinect which measures depth information, etc. A typical application requires inferring the underlying physical phenomenon from data, which is done using machine learning. A fundamental assumption in training models is that the data is Euclidean, i.e. the metric is the standard Euclidean distance governed by the L-2 norm. However in many cases this assumption is violated, when the data lies on non Euclidean spaces such as Riemannian manifolds. While the underlying geometry accounts for the non-linearity, accurate analysis of human activity also requires temporal information to be taken into account. Human movement has a natural interpretation as a trajectory on the underlying feature manifold, as it evolves smoothly in time. A commonly occurring theme in many emerging problems is the need to \emph{represent, compare, and manipulate} such trajectories in a manner that respects the geometric constraints. This dissertation is a comprehensive treatise on modeling Riemannian trajectories to understand and exploit their statistical and dynamical properties. Such properties allow us to formulate novel representations for Riemannian trajectories. For example, the physical constraints on human movement are rarely considered, which results in an unnecessarily large space of features, making search, classification and other applications more complicated. Exploiting statistical properties can help us understand the \emph{true} space of such trajectories. In applications such as stroke rehabilitation where there is a need to differentiate between very similar kinds of movement, dynamical properties can be much more effective. In this regard, we propose a generalization to the Lyapunov exponent to Riemannian manifolds and show its effectiveness for human activity analysis. The theory developed in this thesis naturally leads to several benefits in areas such as data mining, compression, dimensionality reduction, classification, and regression. / Dissertation/Thesis / Doctoral Dissertation Electrical Engineering 2016

Submitted by Willian de Assis Pedrobon Ferreira null (willianferreira51@gmail.com) on 2017-09-27T13:44:04Z No. of bitstreams: 1 dissertacao_Willian_de_Assis_Pedrobon_Ferreira.pdf: 8549439 bytes, checksum: 8a499577dddc476a2f1f7b3cb4d9a873 (MD5) / Approved for entry into archive by Monique Sasaki (sayumi_sasaki@hotmail.com) on 2017-09-28T14:15:16Z (GMT) No. of bitstreams: 1 ferreira_wap_me_ilha.pdf: 8549439 bytes, checksum: 8a499577dddc476a2f1f7b3cb4d9a873 (MD5) / Made available in DSpace on 2017-09-28T14:15:16Z (GMT). No. of bitstreams: 1 ferreira_wap_me_ilha.pdf: 8549439 bytes, checksum: 8a499577dddc476a2f1f7b3cb4d9a873 (MD5) Previous issue date: 2017-08-24 / Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) / A utilização de sensores em ambientes inteligentes é fundamental para supervisionar as atividades dos seres humanos. No reconhecimento de atividades humanas, ou HAR (Human Activity Recognition), técnicas de supervisionamento são aplicadas para identificar as atividades realizadas em diversas aplicações, como no esporte e no acompanhamento de pessoas com necessidades especiais. O Sistema de Reconhecimento de Atividades Humanas (SIRAH) é empregado no reconhecimento de atividades humanas, utilizando um acelerômetro localizado na cintura da pessoa monitorada e uma Rede Neural Artificial para classificar sete atividades: em pé, deitado, sentado, caminhar, correr, sentar e levantar. Originalmente implementado no software MATLAB, realizava classificações offline em que os resultados não eram obtidos durante a execução das atividades. Apresenta-se, neste trabalho, o desenvolvimento de duas versões embarcadas do SIRAH, que executam o algoritmo de classificação durante a prática das atividades monitoradas. A primeira implementação foi efetuada no processador Nios II da Altera, que ofereceu a mesma exatidão do sistema offline com processamento limitado, pois o software consome 673 milissegundos para executar a classificação desejada. Para aprimorar o desempenho, outra versão foi implementada em FPGA utilizando a linguagem de descrição de hardware VHDL. O algoritmo de classificação opera em tempo real e é executado em apenas 236 microssegundos, garantindo total amostragem das acelerações. / The use of sensors in smart environments is fundamental to monitor humans activities. In Human Activity Recognation (HAR), supervision techniques are employed to identify activities in several areas, such as in sport pratice and in people monitoring with special needs. The Sistema de Reconhecimento de Atividades Humanas (SIRAH) is used in human activities recognation, using an accelerometer located on the monitored person waist and an Artificial Neural Network to classify seven activities: standing, lying, seated, walking, running, sitting and standing. Originally, performed offline classifications executed in MATLAB software. In this work we present the development of two embedded SIRAH versions, which perform the classification algorithm during the monitored activities practice. The first implementation was performed on Altera’s Nios II processor, that has been provided the same offline system accuracy, but with limited processing. To improve the performance, the other version was implemented in FPGA using the VHDL hardware description language, which performs real-time classifications, ensuring a lossless acceleration sampling.

abstract: Continuous monitoring of sensor data from smart phones to identify human activities and gestures, puts a heavy load on the smart phone's power consumption. In this research study, the non-Euclidean geometry of the rich sensor data obtained from the user's smart phone is utilized to perform compressive analysis and efficient classification of human activities by employing machine learning techniques. We are interested in the generalization of classical tools for signal approximation to newer spaces, such as rotation data, which is best studied in a non-Euclidean setting, and its application to activity analysis. Attributing to the non-linear nature of the rotation data space, which involve a heavy overload on the smart phone's processor and memory as opposed to feature extraction on the Euclidean space, indexing and compaction of the acquired sensor data is performed prior to feature extraction, to reduce CPU overhead and thereby increase the lifetime of the battery with a little loss in recognition accuracy of the activities. The sensor data represented as unit quaternions, is a more intrinsic representation of the orientation of smart phone compared to Euler angles (which suffers from Gimbal lock problem) or the computationally intensive rotation matrices. Classification algorithms are employed to classify these manifold sequences in the non-Euclidean space. By performing customized indexing (using K-means algorithm) of the evolved manifold sequences before feature extraction, considerable energy savings is achieved in terms of smart phone's battery life. / Dissertation/Thesis / M.S. Electrical Engineering 2014

Submitted by Kevin Quispe (kgmq@icomp.ufam.edu.br) on 2018-10-26T19:02:31Z No. of bitstreams: 1 dissertação-kevin-quispe-final.pdf: 2744401 bytes, checksum: cf4d3337afb0d9fa244abbd4ec3d1a5a (MD5) / Approved for entry into archive by Secretaria PPGI (secretariappgi@icomp.ufam.edu.br) on 2018-10-26T19:07:43Z (GMT) No. of bitstreams: 1 dissertação-kevin-quispe-final.pdf: 2744401 bytes, checksum: cf4d3337afb0d9fa244abbd4ec3d1a5a (MD5) / Approved for entry into archive by Divisão de Documentação/BC Biblioteca Central (ddbc@ufam.edu.br) on 2018-10-26T19:15:25Z (GMT) No. of bitstreams: 1 dissertação-kevin-quispe-final.pdf: 2744401 bytes, checksum: cf4d3337afb0d9fa244abbd4ec3d1a5a (MD5) / Made available in DSpace on 2018-10-26T19:15:25Z (GMT). No. of bitstreams: 1 dissertação-kevin-quispe-final.pdf: 2744401 bytes, checksum: cf4d3337afb0d9fa244abbd4ec3d1a5a (MD5) Previous issue date: 2018-08-14 / Human activity recognition (RAH) through sensors embedded in wearable devices such as smartphones has allowed the development of solutions capable of monitoring human behavior. However, such solutions have presented limitations in terms of efficiency in the consumption of computational resources and generalization for different application or data domain configurations. These limitations are explored in this work in the feature extraction process, in which existing solutions use a manual approach to extract the characteristics of the sensor data. To overcome the problem, this work presents an automatic approach to feature extraction based on the symbolic representation of time series --- representation defined by sets of discrete symbols (words). In this context, this work presents an extension of the symbolic representation of the Bag-Of-SFA-Symbols (BOSS) method to handle the processing of multiple time series, reduce data dimensionality and generate compact and efficient classification models. The proposed method, called Multivariate Bag-Of-SFA-Symbols (MBOSS), is evaluated for the classification of physical activities from data of inertial sensors. Experiments are conducted in three public databases and for different experimental configurations. In addition, the efficiency of the method is evaluated in aspects such as computing time and data space. The results, in general, show an efficiency of classification equivalent to the solutions based on the traditional approach of manual extraction, highlighting the results obtained in the database with nine classes of activities (UniMib SHAR), where MBOSS obtained an accuracy of 99% and 87% for the custom and generalized template, respectively. The efficiency results of MBOSS demonstrate the low computational cost of the solution and show the feasibility of application in smartphones. / O reconhecimento de atividade humanas (RAH) por meio de sensores embutidos em dispositivos vestíveis como, por exemplo, smartphones tem permitido o desenvolvimento de soluções capazes de monitorar o comportamento humano. No entanto, tais soluções têm apresentado limitações em termos de eficiência no consumo dos recursos computacionais e na generalização para diferentes configurações de aplicação ou domínio de dados. Essas limitações são exploradas neste trabalho no processo de extração de características, na qual as soluções existentes utilizam uma abordagem manual para extrair as características dos dados de sensores. Para superar o problema, este trabalho apresenta uma abordagem automática de extração de características baseada na representação simbólica de séries temporais --- representação definida por conjuntos de símbolos discretos (palavras). Nesse contexto, este trabalho apresenta uma extensão do método de representação simbólica Bag-Of-SFA-Symbols (BOSS) para lidar com o processamento de múltiplas séries temporais, reduzir a dimensionalidade dos dados e gerar modelos de classificação compactos e eficiêntes. O método proposto, denominado Multivariate Bag-Of-SFA-Symbols (MBOSS), é avaliado para a classificação de atividades físicas a partir de dados de sensores inerciais. Experimentos são conduzidos em três bases de dados públicas e para diferentes configurações experimentais. Além disso, avalia-se a eficiência do método em aspectos como tempo de computação e espaço de dados. Os resultados, em geral, demostram uma eficácia de classificação equivalente as soluções baseadas na abordagem comun de extração manual de características, destacando os resultados obtidos na base de dados com nove classes de atividades (UniMib SHAR), onde o MBOSS obteve uma acurácia de 99% e 87% para o modelo personalizado e generalizado, respectivamente. Os resultados de eficiência do MBOSS demostram o baixo custo computacional da solução e mostram a viabilidade de aplicação em smartphones.

Description of human activities in videos results not only in detection of actions and objects but also in identification of their active semantic relationships in the scene. Towards this broader goal, we present a combinatorial approach that assumes availability of algorithms for detecting and labeling objects and actions, albeit with some errors. Given these uncertain labels and detected objects, we link them into interpretative structures using domain knowledge encoded with concepts of Grenander’s general pattern theory. Here a semantic video description is built using basic units, termed generators, that represent labels of objects or actions. These generators have multiple out-bonds, each associated with either a type of domain semantics, spatial constraints, temporal constraints or image/video evidence. Generators combine between each other, according to a set of pre-defined combination rules that capture domain semantics, to form larger structures known as configurations, which here will be used to represent video descriptions. Such connected structures of generators are called configurations. This framework offers a powerful representational scheme for its flexibility in spanning a space of interpretative structures (configurations) of varying sizes and structural complexity. We impose a probability distribution on the configuration space, with inferences generated using a Markov Chain Monte Carlo-based simulated annealing algorithm. The primary advantage of the approach is that it handles known computer vision challenges – appearance variability, errors in object label annotation, object clutter, simultaneous events, temporal dependency encoding, etc. – without the need for a exponentially- large (labeled) training data set.

L’objectif principal de cette thèse est de proposer un framework complet pour une découverte, modélisation et reconnaissance automatiques des activités humaines dans les vidéos. Afin de modéliser et de reconnaître des activités dans des vidéos à long terme, nous proposons aussi un framework qui combine des informations perceptuelles globales et locales issues de la scène, et qui construit, en conséquence, des modèles d’activités hiérarchiques. Dans la première catégorie du framework, un classificateur supervisé basé sur le vecteur de Fisher est formé et les étiquettes sémantiques prédites sont intégrées dans les modèles hiérarchiques construits. Dans la seconde catégorie, pour avoir un framework complètement non supervisé, plutôt que d’incorporer les étiquettes sémantiques, les codes visuels formés sont stockés dans les modèles. Nous évaluons les frameworks sur deux ensembles de données réalistes sur les activités de la vie quotidienne enregistrées auprés des patients dans un environnement hospitalier. Pour modéliser des mouvements fins du corps humain, nous proposons quatre différents frameworks de reconnaissance de gestes où chaque framework accepte une ou une combinaison de différentes modalités de données en entrée. Nous évaluons les frameworks développés dans le contexte du test de diagnostic médical, appelé Praxis. Nous proposons un nouveau défi dans la reconnaissance gestuelle qui consiste à obtenir une opinion objective sur les performances correctes et incorrectes de gestes très similaires. Les expériences montrent l’efficacité de notre approche basée sur l’apprentissage en profondeur dans la reconnaissance des gestes et les tâches d’évaluation de la performance. / The main goal of this thesis is to propose a complete framework for automatic discovery, modeling and recognition of human activities in videos. In order to model and recognize activities in long-term videos, we propose a framework that combines global and local perceptual information from the scene and accordingly constructs hierarchical activity models. In the first variation of the framework, a supervised classifier based on Fisher vector is trained and the predicted semantic labels are embedded in the constructed hierarchical models. In the second variation, to have a completely unsupervised framework, rather than embedding the semantic labels, the trained visual codebooks are stored in the models. Finally, we evaluate the proposed frameworks on two realistic Activities of Daily Living datasets recorded from patients in a hospital environment. Furthermore, to model fine motions of human body, we propose four different gesture recognition frameworks where each framework accepts one or combination of different data modalities as input. We evaluate the developed frameworks in the context of medical diagnostic test namely Praxis. Praxis test is a gesture-based diagnostic test, which has been accepted as a diagnostically indicative of cortical pathologies such as Alzheimer’s disease. We suggest a new challenge in gesture recognition, which is to obtain an objective opinion about correct and incorrect performances of very similar gestures. The experiments show effectiveness of our deep learning based approach in gesture recognition and performance assessment tasks.