A testing framework was developed to address system spatial and temporal performance characteristics in a two-dimensional (2D) human motion analysis system using commercially available digital video capture. The first testing protocol involved developing a method to evaluate system spatial performance characteristics with respect to accuracy, precision, and resolution. A physical model comprising a calibration frame was constructed with phantom postures selected to represent joint angles and off-plane movement typical of the activities of interest. This provided reference angles to which angles measured from digitally captured images were compared using the Bland and Altman method. Validation experiments confirmed that the principal sources of error were due to off-plane motion and pixel resolution in the video capture and analysis systems. In these analyses, it was verified that simulated experimental conditions could be corrected using the direct linear transform (DLT); however, the removal of parallax still resulted in 2 degrees of error in measured joint angles. The main source of error was resolution of the data acquisition system verified through Monte Carlo simulations. The second testing protocol involved developing a simple method to determine the temporal accuracy of motion analysis systems incorporating digital video cameras and a pendulum. A planar column pendulum with a natural frequency of 0.872 Hz was used to analyse five systems incorporating commercially available cameras and a single codec. The frame rate for each camera was measured to be within 3% of the US National Television Systems Committee (NTSC) broadcasting digital video standard of 29.97 fps.; however some cameras produced a frame duplication artefact. Least squares curve-fitting using a sinusoidal function revealed RMS differences between 3-5% for angular position and 5-15% for angular speed compared to the captured motion data. It was shown that some digital-video cameras and computer playback software contain data compression technology that may produce substantial temporal frame inaccuracies in recovered video sequences and that temporal accuracy should be evaluated in digital-based human motion analysis systems prior to their use in experimentation. / Thesis (Master, Mechanical and Materials Engineering) -- Queen's University, 2009-08-14 10:54:58.685

This thesis investigates on clinical applicability of a novel sensing technology in the areas of postural steadiness and stroke assessment. The mechanically simple Distributive Tactile Sensing approach is applied to extract motion information from flexible surfaces to identify parameters and disorders of human movement in real time. The thesis reports on the design, implementation and testing of smart platform devices which are developed for discrimination applications through the use of linear and non-linear data interpretation techniques and neural networks for pattern recognition. In the thesis mathematical models of elastic plates, based on finite element and finite difference methods, are developed and described. The models are used to identify constructive parameters of sensing devices by investigating sensitivity and accuracy of Distributive Tactile Sensing surfaces. Two experimental devices have been constructed for the investigation. These are a sensing floor platform for standing applications and a sensing chair for sitting applications. Using a linear approach, the sensing floor platform is developed to detect centre of pressure, an important parameter widely used in the assessment of postural steadiness. It is demonstrated that the locus of centre of pressure can be determined with an average deviation of 1.05mm from that of a commercialised force platform in a balance application test conducted with five healthy volunteers. This amounts to 0.4% of the sensor range. The sensing chair used neural networks for pattern recognition, to identify the level of motor impairment in people with stroke through performing functional reaching task while sitting. The clinical studies with six real stroke survivors have shown the robustness of the sensing technique to deal with a range of possible motion in the reaching task investigated. The work of this thesis demonstrates that the novel Distributive Tactile Sensing approach is suited to clinical and home applications as screening and rehabilitation systems. Mechanical simplicity is a merit of the approach and has potential to lead to versatile low-cost units.

The fundamental task in human motion analysis is the extraction or capture of human motion and the established industrial technique is marker-based human motion capture. However, marker-based systems, apart from being expensive, are obtrusive and require a complex, time-consuming experimental setup, resulting in increased user discomfort. As an alternative solution, research on markerless human motion analysis has increased in prominence. In this thesis, we present three human motion analysis algorithms performing markerless tracking and classification from multiple-view studio-based video sequences using particle swarm optimisation and charting, a subspace learning technique.In our first framework, we formulate, and perform, human motion tracking as a multi-dimensional non-linear optimisation problem, solved using particle swarm optimisation (PSO), a swarm-intelligence algorithm. PSO initialises automatically, does not need a sequence-specific motion model, functioning as a blackbox system, and recovers from tracking divergence through the use of a hierarchical search algorithm (HPSO). We compare experimentally HPSO with particle filter, annealed particle filter and partitioned sampling annealed particle filter, and report similar or better tracking performance. Additionally we report an extensive experimental study of HPSO over ranges of values of its parameters and propose an automatic-adaptive extension of HPSO called as adaptive particle swarm optimisation. Next, in line with recent interest in subspace tracking, where low-dimensional subspaces are learnt from motion models of actions, we perform tracking in a low-dimensional subspace obtained by learning motion models of common actions using charting, a nonlinear dimensionality reduction tool. Tracking takes place in the subspace using an efficient modified version of particle swarm optimisation. Moreover, we perform a fast and efficient pose evaluation by representing the observed image data, multi-view silhouettes, using vector-quantized shape contexts and learning the mapping from the action subspace to shape space using multi-variate relevance vector machines. Tracking results with various action sequences demonstrate the good accuracy and performance of our approach.Finally, we propose a human motion classification algorithm, using charting-based low-dimensional subspaces, to classify human action sub-sequences of varying lengths, or snippets of poses. Each query action is mapped to a single subspace space, learnt from multiple actions. Furthermore we present a system in which, instead of mapping multiple actions to a single subspace, each action is mapped separately to its action-specific subspace. We adopt a multi-layered subspace classification scheme with layered pruning and search. One of the search layers involves comparing the input snippet with a sequence of key-poses extracted from the subspace. Finally, we identify the minimum length of action snippet, of skeletal features, required for classification, using competing classification systems as the baseline. We test our classification component on HumanEva and CMU mocap datasets, achieving similar or better classification accuracy than various comparable systems. human motion and the established industrial technique is marker-based human motion capture. However, marker-based systems, apart from being expensive, are obtrusive and require a complex, time-consuming experimental setup, resulting in increased user discomfort. As an alternative solution, research on markerless human motion analysis has increased in prominence. In this thesis, we present three human motion analysis algorithms performing markerless tracking and clas- si?cation from multiple-view studio-based video sequences using particle swarm optimisation and charting, a subspace learning technique. In our ?rst framework, we formulate, and perform, human motion tracking as a multi-dimensional non-linear optimisation problem, solved using particle swarm optimisation (PSO), a swarm-intelligence algorithm. PSO initialises automat- ically, does not need a sequence-speci?c motion model, functioning as a black- box system, and recovers from temporary tracking divergence through the use of a powerful hierarchical search algorithm (HPSO). We compare experiment- ally HPSO with particle ?lter, annealed particle ?lter and partitioned sampling annealed particle ?lter, and report similar or better tracking performance. Addi- tionally we report an extensive experimental study of HPSO over ranges of values of its parameters and propose an automatic-adaptive extension of HPSO called as adaptive particle swarm optimisation. Next, in line with recent interest in subspace tracking, where low-dimensional subspaces are learnt from motion models of actions, we perform tracking in a low-dimensional subspace obtained by learning motion models of common actions using charting, a nonlinear dimensionality reduction tool. Tracking takes place in the subspace using an e?cient modi?ed version of particle swarm optimisa- tion. Moreover, we perform a fast and e?cient pose evaluation by representing the observed image data, multi-view silhouettes, using vector-quantized shape contexts and learning the mapping from the action subspace to shape space us- ing multi-variate relevance vector machines. Tracking results with various action sequences demonstrate the good accuracy and performance of our approach. Finally, we propose a human motion classi?cation algorithm, using charting-based low-dimensional subspaces, to classify human action sub-sequences of varying lengths, or snippets of poses. Each query action is mapped to a single subspace space, learnt from multiple actions. Furthermore we present a system in which, instead of mapping multiple actions to a single subspace, each action is mapped separately to its action-speci?c subspace. We adopt a multi-layered subspace classi?cation scheme with layered pruning and search. One of the search lay- ers involves comparing the input snippet with a sequence of key-poses extracted from the subspace. Finally, we identify the minimum length of action snippet, of skeletal features, required for accurate classi?cation, using competing classi?ca- tion systems as the baseline. We test our classi?cation component on HumanEva and CMU mocap datasets, achieving similar or better classi?cation accuracy than

Image based motion capture is a problem that has recently gained a lot of attention in the domain of understanding human motion in computer vision. The problem involves estimating the 3D configurations of a human body from a set of images and has applications that include human computer interaction, smart surveillance, video analysis and animation. This thesis takes a machine learning based approach to reconstructing 3D pose and motion from monocular images or video. It makes use of a collection of images and motion capture data to derive mathematical models that allow the recovery of full body configurations directly from image features. The approach is completely data-driven and avoids the use of a human body model. This makes the inference extremely fast. We formulate a class of regression based methods to distill a large training database of motion capture and image data into a compact model that generalizes to predicting pose from new images. The methods rely on using appropriately developed robust image descriptors, learning dynamical models of human motion, and kernelizing the input within a sparse regression framework. Firstly, it is shown how pose can effectively and efficiently be recovered from image silhouettes that are extracted using background subtraction. We exploit sparseness properties of the relevance vector machine for improved generalization and efficiency, and make use of a mixture of regressors for probabilistically handling ambiguities that are present in monocular silhouette based 3D reconstruction. The methods developed enable pose reconstruction from single images as well as tracking motion in video sequences. Secondly, the framework is extended to recover 3D pose from cluttered images by introducing a suitable image encoding that is resistant to changes in background. We show that non-negative matrix factorization can be used to suppress background features and allow the regression to selectively cue on features from the foreground human body. Finally, we study image encoding methods in a broader context and present a novel multi-level image encoding framework called ‘hyperfeatures' that proves to be effective for object recognition and image classification tasks.

Visual analysis of human motion is one of the active research areas in computer vision. The trend shifts from computing motion fields to understanding actions. In this thesis, an action coding scheme based on trajectories of the features calculated with respect to a part based coordinate system is presented. The part based coordinate system is formed using an axis based representation. The features are extracted from images segmented in the form of silhouettes. We present some preliminary experiments that demonstrate the potential of the method in action similarity analysis.

Orientador: Prof. Dr. Marcos Duarte / Dissertação (mestrado) - Universidade Federal do ABC. Programa de Pós-Graduação em Engenharia Biomédica, 2016. / Atualmente os dispositivos de instrumentação biomecânica, sobretudo os sistemas de captura de movimento, são capazes de fornecer um grande número de dados. Seja para extrair informações para análise de diferenças entre grupos, para semi-automatização de procedimentos clínicos típicos de laboratórios de marcha ou para o desenvolvimento de sistemas inteligentes que fazem uso da informação do movimento, o reconhecimento de padrões de movimento é uma necessidade. Estes dados podem ser reduzidos a séries temporais, sendo assim estamos diante de um problema de mineração de dados de séries temporais. No entanto enquanto a maior parte das pesquisas se concentram em tarefas de mineração, o problema fundamental de como representar uma série temporal ainda não foi plenamente abordado até agora. Este projeto objetiva estudar representações simbólicas para séries temporais provenientes de dados de captura de movimentos, uma abordagem ainda muito pouco explorada. Propondo a elaboração de um algoritmo que realiza o mapeamento entre um conjunto de séries temporais e uma sequência de símbolos, levando em consideração informações do domínio da biomecânica e controle motor. Um conjunto de movimentos discretos foram convertidos em uma representação simbólica, a partir da qual foi realizado um agrupamento hierárquico e classificação em 10 atividades rotuladas, com uma exatidão de 84.72% e um padrão sequencial foi detectado no andar. Estes resultados foram obtidos em um tempo de processamento relativamente baixo e a partir de apenas 3 ângulos no plano sagital do membro inferior direito. Com este trabalho atingimos o objetivo proposto e estabelecemos uma representação simbólica do movimento, denominada palavras do movimento, com boa parte das características almejadas. É uma representação simples e prática, a partir da qual foi possível estabelecer uma métrica que quantifica a similaridade entre movimentos. / Currently biomechanical instrumentation devices, especially motion capture systems, are able to provide a large amount of data. Be it for extract information to compare differences among different groups, for semi-automation of typical clinical gait analysis procedures or for developing intelligent systems that make use of motion information, recognition of motion patterns is a need. These data can be reduced to time series, so we are facing a problem of mining time series data. However, while most of the research communities have concentrated on the mining tasks, the fundamental problem on how to represent a time series has not yet been fully addressed so far. This project aims to study symbolic representation for time series data from motion capture, an approach still not much explored. The development of the algorithm that performs the mapping between a set of time series and a sequence of symbols, taking into account information from the field of biomechanics and motor control. A set of discrete motions were converted to a symbolic representation, from which we performed a hierarchical clustering and classified in 10 labeled activities, with an accuracy of 84.72% and a sequential pattern was detected in gait. These results were obtained at a relatively low processing time and from only 3 angles in the sagittal plane of the right lower limb. We achieved our objective and established a symbolic representation of the movement, called motion words, with most of it desired characteristics. It is a simple and practical representation, from which it was possible establish a metric that quantifies the similarity between movements.

Face aux limites auxquelles doivent faire face les systèmes optoélectroniques (matériel lourd, champ de mesure limité), les capteurs inertiels constituent une alternative prometteuse pour la mesure du mouvement humain. Grâce aux dernières avancées techniques, notamment en termes de miniaturisation des capteurs, leur utilisation en ambulatoire c’est-à-dire de façon autonome et embarquée est devenue possible. Mais ces opérations de miniaturisation ne sont pas sans effet sur les performances de ces capteurs. En effet, une telle mesure est dégradée par différents types de perturbations (stochastiques et déterministes) qui sont alors propagées au cours du processus dit de fusion des données visant à estimer l'orientation des segments humains. Classiquement, cette opération est réalisée à l'aide d'un filtre de Kalman dont le rôle est justement d'estimer une grandeur à partir d'une mesure bruitée en la confrontant à un modèle d'évolution.Dans ce contexte, nous proposons diverses méthodologies dans le but d'accéder à une mesure suffisamment précise pour être exploitée dans le cadre de l'analyse du mouvement humain. La première partie de cette thèse se focalise sur les capteurs. Tout d'abord, nous étudions les bruits de mesure issus des capteurs inertiels, puis nous leur attribuons un modèle afin de les prendre en compte au sein du filtre de Kalman. Ensuite, nous analysons les procédures de calibrage et évaluons leurs effets réels sur la mesure afin d'émettre quelques propositions en termes de compromis performance/facilité de réalisation.Dans une seconde partie, nous nous consacrons à l'algorithme de fusion des données. Après avoir proposé un filtre de Kalman adapté à la mesure du mouvement humain, nous nous focalisons sur un problème récurrent à ce stade : l'identification des matrices de covariance dont le rôle est d'attribuer une caractérisation globale aux erreurs de mesure. Cette méthode, basée sur une confrontation de la mesure avec une référence issue d'un système optoélectronique, met en évidence la nécessité de traiter ce problème rigoureusement.Dans une troisième partie, nous commençons à aborder les problèmes liés à l'utilisation des capteurs inertiels pour la mesure du mouvement humain, notamment le calibrage anatomique et le positionnement des capteurs.En conclusion, les gains apportés par les diverses propositions avancées dans cette thèse sont évalués et discutés. / To face the limits of optoelectronic systems (heavy device, restricted measurement field), inertial sensors are a promising alternative for human motion analysis. Thanks to the latest technical advancements like sensor miniaturization, they can now work autonomously which makes possible to directly embed them on the human segments. But, as a counterpart of these developments, inertial sensor measurement still suffers from both stochastic and deterministic perturbations. The induced errors then propagate over the so-called fusion algorithm used to estimate human segment orientation. A common tool to perform such an operation is the Kalman filter that estimates unknown variables by correcting noisy measurements by the use of a dynamic model.With the aim of achieving a sufficiently accurate measurement to perform human motion analysis, various methodologies are proposed in the present work. The first part of this thesis focuses on the sensors. First, inertial sensor noises are studied and modeled in order to be integrated into the Kalman filter. Calibration processes as their effects over the measurement are for that purposed analyzed. Some recommendations are thus proposed to reach a compromise between calibration performance and complexity.In a second part, the data fusion algorithm is approached. A specific Kalman filter dedicated to human motion measurement is first proposed. Then, a recurrent problem is studied in details: the definition of the covariance matrix that represents a globalcharacterization of the measurement errors. Considering an optoelectronic system as a reference to compare inertial measurement, a method is proposed for this covariance matrix identification, which also highlights the need to address this problem rigorously.In a third part, we begin to address the use of inertial sensors for human motion analysis by focusing on models and IMU-to-segment calibration.To conclude, the benefits made by the proposed methodologies are evaluated and discussed.

Human motion analysis is one of the major problems in computer vision research. It deals with the study of the motion of human body in video data from different aspects, ranging from the tracking of body parts and reconstruction of 3D human body configuration, to higher level of interpretation of human action and activities in image sequences. When human motion is observed through video camera, it is perspectively distorted and may appear totally different from different viewpoints. Therefore it is highly challenging to establish correct relationships between human motions across video sequences with different camera settings. In this work, we investigate the geometric invariance in the motion of human body, which is critical to accurately understand human motion in video data regardless of variations in camera parameters and viewpoints. In human action analysis, the representation of human action is a very important issue, and it usually determines the nature of the solutions, including their limits in resolving the problem. Unlike existing research that study human motion as a whole 2D/3D object or a sequence of postures, we study human motion as a sequence of body pose transitions. We also decompose a human body pose further into a number of body point triplets, and break down a pose transition into the transition of a set of body point triplets. In this way the study of complex non-rigid motion of human body is reduced to that of the motion of rigid body point triplets, i.e. a collection of planes in motion. As a result, projective geometry and linear algebra can be applied to explore the geometric invariance in human motion. Based on this formulation, we have discovered the fundamental ratio invariant and the eigenvalue equality invariant in human motion. We also propose solutions based on these geometric invariants to the problems of view-invariant recognition of human postures and actions, as well as analysis of human motion styles. These invariants and their applicability have been validated by experimental results supporting that their effectiveness in understanding human motion with various camera parameters and viewpoints.

L'objectif de ce travail est de développer des méthodes avancées d'apprentissage pour l’analyse et l'interprétation automatique du mouvement humain à partir de sources d'information diverses, telles que les images, les vidéos, les cartes de profondeur, les données de type “MoCap” (capture de mouvement), les signaux audio et les données issues de capteurs inertiels. A cet effet, nous proposons plusieurs modèles neuronaux et des algorithmes d’entrainement associés pour l’apprentissage supervisé et semi-supervisé de caractéristiques. Nous proposons des approches de modélisation des dépendances temporelles, et nous montrons leur efficacité sur un ensemble de tâches fondamentales, comprenant la détection, la classification, l’estimation de paramètres et la vérification des utilisateurs (la biométrie). En explorant différentes stratégies de fusion, nous montrons que la fusion des modalités à plusieurs échelles spatiales et temporelles conduit à une augmentation significative des taux de reconnaissance, ce qui permet au modèle de compenser les erreurs des classifieurs individuels et le bruit dans les différents canaux. En outre, la technique proposée assure la robustesse du classifieur face à la perte éventuelle d’un ou de plusieurs canaux. Dans un deuxième temps nous abordons le problème de l’estimation de la posture de la main en présentant une nouvelle méthode de régression à partir d’images de profondeur. Dernièrement, dans le cadre d’un projet séparé (mais lié thématiquement), nous explorons des modèles temporels pour l'authentification automatique des utilisateurs de smartphones à partir de leurs habitudes de tenir, de bouger et de déplacer leurs téléphones. Dans ce contexte, les données sont acquises par des capteurs inertiels embraqués dans les appareils mobiles. / The research goal of this work is to develop learning methods advancing automatic analysis and interpreting of human motion from different perspectives and based on various sources of information, such as images, video, depth, mocap data, audio and inertial sensors. For this purpose, we propose a several deep neural models and associated training algorithms for supervised classification and semi-supervised feature learning, as well as modelling of temporal dependencies, and show their efficiency on a set of fundamental tasks, including detection, classification, parameter estimation and user verification. First, we present a method for human action and gesture spotting and classification based on multi-scale and multi-modal deep learning from visual signals (such as video, depth and mocap data). Key to our technique is a training strategy which exploits, first, careful initialization of individual modalities and, second, gradual fusion involving random dropping of separate channels (dubbed ModDrop) for learning cross-modality correlations while preserving uniqueness of each modality-specific representation. Moving forward, from 1 to N mapping to continuous evaluation of gesture parameters, we address the problem of hand pose estimation and present a new method for regression on depth images, based on semi-supervised learning using convolutional deep neural networks, where raw depth data is fused with an intermediate representation in the form of a segmentation of the hand into parts. In separate but related work, we explore convolutional temporal models for human authentication based on their motion patterns. In this project, the data is captured by inertial sensors (such as accelerometers and gyroscopes) built in mobile devices. We propose an optimized shift-invariant dense convolutional mechanism and incorporate the discriminatively-trained dynamic features in a probabilistic generative framework taking into account temporal characteristics. Our results demonstrate, that human kinematics convey important information about user identity and can serve as a valuable component of multi-modal authentication systems.

Estimating human motion is the topic of this thesis. We are interested in accurately estimating the motion of a human body using only video images capturing the subject in motion. Video images from up to two cameras are considered. The first main topic of the thesis is to investigate a new type of input data. This data consists of some sort of texture. This texture can be added to the human body segment under study or it can be the actual texture of the skin. In paper I we investigate if added texture together with the use of a two camera system can provide enough information to make it possible to estimate the knee joint center location. Evaluation is made using a marker based system that is run in parallel to the two camera video system. The results from this investigation show promise for the use of texture. The marker and texture based estimates differ in absolute values but the variations are similar indicating that texture is in fact usable for this purpose. In paper II and III we investigate further the usability in images of skin texture as input for motion estimation. Paper II approaches the problem of estimating human limb motion in the image plane. An image histogram based mutual information criterion is used to decide if an extracted image patch from frame k is a good match to some location in frame k+1. Eval- uation is again performed using a marker based system synchronized to the video stream. The results are very promising for the application of skin texture based motion estimation in 2D. In paper III, basically the same approach is taken as in paper II with the substantial difference that here estimation of three dimensional motion is addressed. Two video cameras are used and the image patch matching is performed both between cameras (inter-camera) in frame k and also in each cameras images (intra-camera) for frame k to k+1. The inter-camera matches yield triangulated three dimensional estimates on the approximate surface of the skin. The intra-camera matches provide a way to connect the three dimensional points between frame k and k+1 The resulting one step three dimensional trajectories are then used to estimate rigid body motion using least squares methods. The results show that there is still some work to be done before this texture based method can be an alternative to the marker based methods. In paper IV the second main topic of the thesis is discussed. Here we present an investigation in using model based techniques for the purpose of estimating human motion. A kinematic model of the thigh and shank segments are built with an anatomic model of the knee. Using this model, the popular particle filter and typical simulated data from the triangulation in paper III, an estimate of the motion variables in the thigh and shank segment can be achieved. This also includes one static model parameter used to describe the knee model. The results from this investigation show good promise for the use of triangulated skin texture as input to such a model based approach.