Augmented Reality (AR) has the possibility of interacting with virtual objects and real objects at the same time since it combines the real world with computer-generated contents seamlessly. However, most AR interface research uses general Virtual Reality (VR) interaction techniques without modification. In this research we develop a multimodal interface (MMI) for AR with speech and 3D hand gesture input. We develop a multimodal signal fusion architecture based on the user behaviour while interacting with the MMI that provides more effective and natural multimodal signal fusion. Speech and 3D vision-based free hand gestures are used as multimodal input channels. There were two user observations (1) a Wizard of Oz study and (2)Gesture modelling. With the Wizard of Oz study, we observed user behaviours of interaction with our MMI. Gesture modelling was undertaken to explore whether different types of gestures can be described by pattern curves. Based on the experimental observations, we designed our own multimodal fusion architecture and developed an MMI. User evaluations have been conducted to evaluate the usability of our MMI. As a result, we found that MMI is more efficient and users are more satisfied with it when compared to the unimodal interfaces. We also describe design guidelines which were derived from our findings through the user studies.

The design discipline is reliant on communication as a means to express and share ideas during the creation of products. Design communication can take place in two distinct settings: formal communication presentations and informal interaction communication. Group design communication involves the communication of various attributes of a design. Attributes such as the scale, shape and functionality of a product are communicated. Designers employ various mechanisms including verbal, non-verbal (e.g. hand gestures), and physical tools (e.g. sketches and models) to communicate attributes. Although the verbal and physical tools are commonly used effective tools, hand gestures remain underused. Yet, hand gestures are extensions of the human mind, which may reveal thoughts that verbal communication may not be able to communicate (McNeil, 1992). Hand gestures have the potential to reveal thoughts and if used as an interface mechanism, can augment the information space. As such, hand gestures may facilitate design communication within a group. This study explores the potential of hand gestures as tools in design communication. Moreover; the purpose of this study is to understand how gestures can facilitate the design communication that occurs within a group. A two-phase study was proposed: a research phase and a design phase. During the research phase, ethnographic research in design education environments was conducted to understand what gestures are produced and how they are used during formal and informal design communication settings. The data collected was analyzed and categorized to reveal quantitative and qualitative results. Survey studies were also conducted to validate the hand gesture meaning. The data was used to create design guidelines that directed the design phase, where concepts used hand gestures as interface mechanisms, to augment the experience of a design information exchange. The significance of this project is to generate new knowledge to be applied to the development of more natural technological systems where gestures are used as an alternative to current input devices (e.g. mouse and keyboard) for navigation and manipulation of design material amongst a group. The goal was to improve the communication between designers and their materials and enhance the experience of distributing and receiving design information.

Thesis (Ph. D.)--University of Chicago, Dept. of Psychology, August 1999. / Includes bibliographical references. Also available on the Internet.

Submitted by Glauber Assunção Moreira (glauber.a.moreira@gmail.com) on 2018-08-22T16:02:22Z No. of bitstreams: 3 1.ELEMENTOS PRÉ-TEXTUAIS.doc: 72704 bytes, checksum: cae1472cf9c093ffe6c5f283b3e2bf09 (MD5) 2.APRESENTAÇÃO ATÉ CONSID. FINAIS.doc: 1522688 bytes, checksum: 87bede749495c8767b852e7a501161b2 (MD5) 3.ELEMENTOS PÓS-TEXTUAIS.doc: 90112 bytes, checksum: f34f912011209fcdce76ebda3363ad0b (MD5) / Approved for entry into archive by Ednaide Gondim Magalhães (ednaide@ufba.br) on 2018-08-23T11:56:45Z (GMT) No. of bitstreams: 3 1.ELEMENTOS PRÉ-TEXTUAIS.doc: 72704 bytes, checksum: cae1472cf9c093ffe6c5f283b3e2bf09 (MD5) 2.APRESENTAÇÃO ATÉ CONSID. FINAIS.doc: 1522688 bytes, checksum: 87bede749495c8767b852e7a501161b2 (MD5) 3.ELEMENTOS PÓS-TEXTUAIS.doc: 90112 bytes, checksum: f34f912011209fcdce76ebda3363ad0b (MD5) / Made available in DSpace on 2018-08-23T11:56:45Z (GMT). No. of bitstreams: 3 1.ELEMENTOS PRÉ-TEXTUAIS.doc: 72704 bytes, checksum: cae1472cf9c093ffe6c5f283b3e2bf09 (MD5) 2.APRESENTAÇÃO ATÉ CONSID. FINAIS.doc: 1522688 bytes, checksum: 87bede749495c8767b852e7a501161b2 (MD5) 3.ELEMENTOS PÓS-TEXTUAIS.doc: 90112 bytes, checksum: f34f912011209fcdce76ebda3363ad0b (MD5) / RESUMO Esta dissertação consiste em uma análise estética e reflexiva dos processos de criação e encenação do espetáculo coreográfico Metrópole, criado e dirigido pela autora, que aqui se apresenta, e interpretado pela Companhia Moderno de Dança. Dentre seus objetivos destacam-se: analisar o processo de abstração cênica do cotidiano urbano, incluindo o percurso do gesto, da sua forma prática e cotidiana à sua forma artística, compreendendo, através de teorias da estética, a função desse gesto em ambas as formas; identificar, a partir de um breve enfoque antropológico, a origem e a presença da gestualidade cotidiana abstraída no espetáculo, refletindo sua utilização na coreografia do mesmo; verificar as características da pós-modernidade coreográfica no espetáculo e fazer um registro do espetáculo analisado para posteriores utilizações. A pesquisa indicou que a presença do cotidiano se notifica no espetáculo, não unicamente a partir da observação de gestualidades cotidianas visivelmente presentes nas grandes cidades, mas também a partir da pesquisa e criação de movimentos representativos dos sentimentos que se estabelecem no jogo do convívio social metropolitano, além de ter sido confirmada, tanto na criação quanto na encenação do espetáculo, a presença e a continuidade do gesto cotidiano comum. Procurando compreender essa permanência, foi criado o conceito de impregnação cultural, isto é, a maneira como as características humanas de comportamento gestual cotidiano são apreendidas e verificadas no corpo. Além disso, foi detectada ainda a presença de diversos elementos caracterizadores da pós-modernidade na dança, os quais, de forma eminentemente reflexiva, encontram-se descritos ao longo do texto. / ABSTRACT This dissertation consists in an aesthetics and reflexive analysis of the creation and staging processes of the choreografic spectacle Metrópole, created and directed by the author that here is presenting herself and performed by the Companhia Moderno de Dança. Among its objectives we point out: to analyse the abstraction processe from the urban quotidian to the scene, including the gesture trajectory from its practical and quotidian form to its artistic form, understanding both functions through the aesthetics theories; to identify the permanence of the quotidian gesture in the choreography, even in front of the gesture’s artistic transformation, through a brief antropological focus; to verify the postmodern dance characteristics in the spectacle, signed for me as my first one in this style of dancing and to make one spectacle’s registration for future utilization. The research pointed that the common gesture is found in the spectacle not only from the quotidian gestualities observation perceptible in the big cities, but also from research and movement creation that represent feelings of the metropolitan social living game. Besides, the presence and cotinuity of the common and quotidian gesture was confirmed in both the creation and staging processes. Looking for understing this we created the concept of cultural impregnation, that means the way as the human characteristics are apprehended and verified in the body. There were also verified the presence of diferent postmodern dance characteristics, elements wich are descripted along the text by a reflexive manner.

Gesture recognition plays an important role in human computer Interaction for intelligent computing. Major applications like Gaming, Robotics and Automated Homes uses gesture recognition techniques which diminishes the usage of mechanical devices. The main goal of my thesis is to interpret SWAT team gestures using different types of sensors. Accelerometer and flex sensors were explored extensively to build a prototype for soldiers to communicate in the absence of line of sight. Arm movements were recognized by flex sensors and motion gestures by Accelerometers. Accelerometers are used to measure acceleration in respect to movement of the sensor in 3D. Flex sensors changes its resistance based on the amount of bend in the sensor. SVM is the classification algorithm used for classification of the samples. LIBSVM (Library for Support Vector Machines) is integrated software for support vector classification, regression and distribution estimation which supports multi class classification. Sensors data is connected to the WI micro dig to digitize the signal and to transmit it wirelessly to the computing device. Feature extraction and Signal windowing were the two major factors which contribute for the accuracy of the system. Mean Average value and Standard Deviation are the two features considered for accelerometer sensor data classification and Standard deviation is used for the flex sensor analysis for optimum results. Filtering of the signal is done by identifying the different states of signals which are continuously sampled.

Recentemente, tem se percebido um interesse maior da área de computação pela pesquisa em análise de gestos. Parte dessas pesquisas visa dar suporte aos pesquisadores da área de \"estudos dos gestos\", que estuda o uso de partes do corpo para fins comunicativos. Pesquisadores dessa área analisam gestos a partir de transcrições de conversas ou discursos gravados em vídeo. Para a transcrição dos gestos, geralmente realiza-se a sua segmentação em unidades gestuais e fases. O presente trabalho tem por objetivo desenvolver estratégias para segmentação automatizada das unidades gestuais e das fases dos gestos contidos em um vídeo no contexto de contação de histórias, formulando o problema como uma tarefa de classificação supervisionada. As Máquinas de Vetores Suporte foram escolhidas como método de classificação, devido à sua capacidade de generalização e aos bons resultados obtidos para diversos problemas complexos. Máquinas de Vetores Suporte, porém, não consideram os aspectos temporais dos dados, características que são importantes na análise dos gestos. Por esse motivo, este trabalho investiga métodos de representação temporal e variações das Máquinas de Vetores Suporte que consideram raciocínio temporal. Vários experimentos foram executados neste contexto para segmentação de unidades gestuais. Os melhores resultados foram obtidos com Máquinas de Vetores Suporte tradicionais aplicadas a dados janelados. Além disso, três estratégias de classificação multiclasse foram aplicadas ao problema de segmentação das fases dos gestos. Os resultados indicam que um bom desempenho para a segmentação de gestos pode ser obtido ao realizar o treinamento da estratégia com um trecho inicial do vídeo para obter uma segmentação automatizada do restante do vídeo. Assim, os pesquisadores da área de \"estudos dos gestos\" poderiam segmentar manualmente apenas um trecho do vídeo, reduzindo o tempo necessário para realizar a análise dos gestos presentes em gravações longas. / Recently, it has been noted an increasing interest from computer science for research on gesture analysis. Some of these researches aims at supporting researchers from \"gesture studies\", which studies the use of several body parts for communicative purposes. Researchers of \"gesture studies\" analyze gestures from transcriptions of conversations and discourses recorded in video. For gesture transcriptions, gesture unit segmentation and gesture phase segmentation are usually employed. This study aims to develop strategies for automated segmentation of gestural units and phases of gestures contained in a video in the context of storytelling, formulating the problem as a supervised classification task. Support Vector Machines were selected as classification method, because of its ability to generalize and good results obtained for many complex problems. Support Vector Machines, however, do not consider the temporal aspects of data, characteristics that are important for gesture analysis. Therefore, this paper investigates methods of temporal representation and variations of the Support Vector machines that consider temporal reasoning. Several experiments were performed in this context for gesture units segmentation. The best results were obtained with traditional Support Vector Machines applied to windowed data. In addition, three strategies of multiclass classification were applied to the problem of gesture phase segmentation. The results indicate that a good performance for gesture segmentation can be obtained by training the strategy with an initial part of the video to get an automated segmentation of the rest of the video. Thus, researchers in \"gesture studies\" could manually segment only part of the video, reducing the time needed to perform the analysis of gestures contained in long recordings.

Recentemente, tem se percebido um interesse maior da área de computação pela pesquisa em análise de gestos. Parte dessas pesquisas visa dar suporte aos pesquisadores da área de \"estudos dos gestos\", que estuda o uso de partes do corpo para fins comunicativos. Pesquisadores dessa área analisam gestos a partir de transcrições de conversas ou discursos gravados em vídeo. Para a transcrição dos gestos, geralmente realiza-se a sua segmentação em unidades gestuais e fases. O presente trabalho tem por objetivo desenvolver estratégias para segmentação automatizada das unidades gestuais e das fases dos gestos contidos em um vídeo no contexto de contação de histórias, formulando o problema como uma tarefa de classificação supervisionada. As Máquinas de Vetores Suporte foram escolhidas como método de classificação, devido à sua capacidade de generalização e aos bons resultados obtidos para diversos problemas complexos. Máquinas de Vetores Suporte, porém, não consideram os aspectos temporais dos dados, características que são importantes na análise dos gestos. Por esse motivo, este trabalho investiga métodos de representação temporal e variações das Máquinas de Vetores Suporte que consideram raciocínio temporal. Vários experimentos foram executados neste contexto para segmentação de unidades gestuais. Os melhores resultados foram obtidos com Máquinas de Vetores Suporte tradicionais aplicadas a dados janelados. Além disso, três estratégias de classificação multiclasse foram aplicadas ao problema de segmentação das fases dos gestos. Os resultados indicam que um bom desempenho para a segmentação de gestos pode ser obtido ao realizar o treinamento da estratégia com um trecho inicial do vídeo para obter uma segmentação automatizada do restante do vídeo. Assim, os pesquisadores da área de \"estudos dos gestos\" poderiam segmentar manualmente apenas um trecho do vídeo, reduzindo o tempo necessário para realizar a análise dos gestos presentes em gravações longas. / Recently, it has been noted an increasing interest from computer science for research on gesture analysis. Some of these researches aims at supporting researchers from \"gesture studies\", which studies the use of several body parts for communicative purposes. Researchers of \"gesture studies\" analyze gestures from transcriptions of conversations and discourses recorded in video. For gesture transcriptions, gesture unit segmentation and gesture phase segmentation are usually employed. This study aims to develop strategies for automated segmentation of gestural units and phases of gestures contained in a video in the context of storytelling, formulating the problem as a supervised classification task. Support Vector Machines were selected as classification method, because of its ability to generalize and good results obtained for many complex problems. Support Vector Machines, however, do not consider the temporal aspects of data, characteristics that are important for gesture analysis. Therefore, this paper investigates methods of temporal representation and variations of the Support Vector machines that consider temporal reasoning. Several experiments were performed in this context for gesture units segmentation. The best results were obtained with traditional Support Vector Machines applied to windowed data. In addition, three strategies of multiclass classification were applied to the problem of gesture phase segmentation. The results indicate that a good performance for gesture segmentation can be obtained by training the strategy with an initial part of the video to get an automated segmentation of the rest of the video. Thus, researchers in \"gesture studies\" could manually segment only part of the video, reducing the time needed to perform the analysis of gestures contained in long recordings.

Dans cette thèse, nous nous intéressons à la reconnaissance de l'activité humaine. Nous commençons par proposer notre propre définition d'une action : une action est une séquence prédéfinie de gestes simples et concaténés. Ainsi, des actions similaires sont composées par les mêmes gestes simples. Chaque réalisation d'une action (enregistrement) est unique. Le corps humain et ses articulations vont effectuer les mêmes mouvements que celles d'un enregistrement de référence, avec des variations d'amplitude et de dynamique ne devant pas dépasser certaines limites qui conduiraient à un changement complet d'action. Pour effectuer nos expérimentations, nous avons capturé un jeu de données contenant des variations de base, puis fusionné certains enregistrements avec d'autres actions pour former un second jeu induisant plus de confusion au cours de la classification. Ensuite, nous avons capturé trois autres jeux contenant des propriétés intéressantes pour nos expérimentations avec la Fusion Tardive Asynchrone (ou Asynchronous Late Fusion notée ALF). Nous avons surmonté le problème des petits jeux non discriminants pour la reconnaissance d'actions en étendant un ensemble d'enregistrements effectués par différentes personnes et capturés par une caméra RGB-D. Nous avons présenté une nouvelle méthode pour générer des enregistrements synthétiques pouvant être utilisés pour l'apprentissage d'algorithmes de reconnaissance de l'activité humaine. La méthode de simulation a ainsi permis d'améliorer les performances des différents classifieurs. Un aperçu général de la classification des données dans un contexte audiovisuel a conduit à l'idée de l'ALF. En effet, la plupart des approches dans ce domaine classifient les flux audio et vidéo séparément, avec des outils différents. Chaque séquence temporelle est analysée séparément, comme dans l'analyse de flux audiovisuels, où la classification délivre des décisions à des instants différents. Ainsi, pour déduire la décision finale, il est important de fusionner les décisions prises séparément, d'où l'idée de la fusion asynchrone. Donc, nous avons trouvé intéressant d'appliquer l'ALF à des séquences temporelles. Nous avons introduit l'ALF afin d'améliorer la classification temporelle appliquée à des algorithmes de fusion tardive tout en justifiant l'utilisation d'un modèle asynchrone lors de la classification des données temporelles. Ensuite, nous avons présenté l'algorithme de l'ALF et les paramètres utilisés pour l'optimiser. Enfin, après avoir mesuré les performances de classifications avec différents algorithmes et jeux de données, nous avons montré que l'ALF donne de meilleurs résultats qu'une solution synchrone simple. Etant donné qu'il peut être difficile d'identifier les jeux de données compatibles avec l'ALF, nous avons construit des indicateurs permettant d'en extraire des informations statistiques. / In this thesis, we took interest in human action recognition. Thus, it was important to define an action. We proposed our own definition: an action is a predefined sequence of concatenated simple gestures. The same actions are composed of the same simple gestures. Every performance of an action (recording) is unique. Hence, the body and the joints will perform the same movements as the reference recording, with changes of dynamicity of the sequence and amplitude in the DOF. We note that the variations in the amplitude and dynamicity must not exceed certain boundaries in order not to lead to entirely different actions. For our experiments, we captured a dataset composed of actions containing basic variations. We merged some of those recordings with other actions to form a second dataset, consequently inducing more confusion than the previous one during the classification. We also captured three other datasets with properties that are interesting for our experimentations with the ALF (Asynchronous Late Fusion). We overcame the problem of non-discriminatory actions datasets for action recognition by enlarging a set of recordings performed by different persons and captured by an RGB-D camera. We presented a novel method for generating synthetic recordings, for training action recognition algorithms. We analyzed the parameters of the method and identified the most appropriate ones, for the different classifiers. The simulation method improved the performances while classifying different datasets. A general overview of data classification starting from the audio-visual context led to the ALF idea. In fact, most of the approaches in the domain classify sound and video streams separately with different tools. Every temporal sequence from a recording is analyzed distinctly, as in audiovisual stream analysis, where the classification outputs decisions at various time instants. Therefore, to infer the final decision, it is important to fuse the decisions that were taken separately, hence the idea of the asynchronous fusion. As a result, we found it interesting to implement the ALF in temporal sequences. We introduced the ALF model for improving temporal events classification applied on late fusion classification algorithms. We showed the reason behind the use of an asynchronous model when classifying datasets with temporal properties. Then, we introduced the algorithm behind the ALF and the parameters used to tune it. Finally, according to computed performances from different algorithms and datasets, we showed that the ALF improves the results of a simple Synchronous solution in most of the cases. As it can be difficult for the user of the ALF solution to determine which datasets are compatible with the ALF, we built indicators to compare the datasets by extracting statistical information from the recordings. We developed indexes: the ASI and the ASIP, combined into a final index (the ASIv) to provide information concerning the compatibility of the dataset with the ALF. We evaluated the performances of the ALF on the segmentation of action series and compared the results between synchronous and ALF solutions. The method that we proposed increased the performances. We analyzed the human movement and gave a general definition of an action. Later, we improved this definition and proposed a "visual definition" of an action. With the aid of the ALF model, we focus on the parts and joints of an action that are the most discriminant and display them in an image. In the end, we proposed multiple paths as future studies. The most important ones are : - Working on a process to find the ALF's number of parts using the ASIv. - Reducing the complexity by finding the discriminant joints and features thanks to the ALF properties - Studying the MD-DTW features in-depth since the algorithm depends on the choice of the features - Implementing a DNN for comparison purposes - Developing the confidence coefficient.

Constructivism remains one of the most influential views of understanding how children learn science today. Research investigating learning from within this viewpoint has led to the development of a range of theoretical models, most of which aim to explain the underlying processes associated with conceptual change. Such models range in depth and scope with some attributing change to purely cognitive processes while others suggest a role for social factors. Contemporary research has also begun to explore links between the role of practical activity, skills development and language. This study utilises a cross-sectional design in order to investigate the development of children’s ideas and concepts related to two areas of the English National Curriculum for Science: ‘electricity’ and ‘floating and sinking’. A new and innovative multimodal methodology combining practical science activities and traditional / conventional perspectives alongside interview and observational protocols is presented. Multimodal research proposes that knowledge and meaning are transmitted through a range of responses types including language, drawings and gesture. The participants in this study were children aged 7, 11 and 14 years attending four schools in the East Midlands region. Results demonstrate that the children’s ideas could be developed using conceptual challenge tasks. The gestures that the children produced were categorised according to five different forms: referential, representative, expressive, thinking and social, often containing information about their science ideas that was not included in other response types. The results also begin to uncover how meaning is socially constructed and supported. These results form the basis of a critique of methodology intended to re-evaluate and inform debate arising from different models of conceptual change. The potential importance of studying children’s gestures in classroom settings for providing important cues and clues to underlying thoughts that may not be present in verbal or other more conventional responses alone is highlighted.

Thesis (MA)--Stellenbosch University, 2015. / ENGLISH ABSTRACT: This thesis aims to examine to what extent the visual representations of ancient Mesopotamia portrayed the royal ideology that was present during the time of their intended display. The iconographic method is used in this study and this allows for a better understanding of the meaning behind the work of art. This method allows the study to better attempt to comprehend the underlying ideology of the work of art. The eight images studied date between three thousand BCE and one thousand BCE and this provides a broad base for the study. By having such a broad base it enables the study to provide a brief understanding of how the ideology adapted over two thousand years. The broad base also enables the study to examine a variety of different gestures that are portrayed on the representations. This thereby provides the reader with a better understanding of why certain gestures were used and how the underlying ideology was communicated through these movements. The study concludes that while the gestures lend a life-like appearance to the representation they do not solely portray an underlying ideological message. Rather, they enhance the already inherent ideological message. / AFRIKAANSE OPSOMMING: Hierdie tesis ondersoek tot watter mate die visuele voorstellings van Ou Mesopotamië die koninklike ideologie — van die tyd toe hulle uitgestal is — uitgebeeld het. Die ikonografiese metode is in hierdie studie gebruik en maak dit moontlik om 'n beter begrip van die betekenis agter die kunswerk te verkry. Die metode stel die studie in staat om die onderliggende ideologie van die kunswerk beter te verstaan. Die agt bestudeerde beelde dateer tussen drieduisend v.C. en 'n duisend v.C. en bied 'n breë basis vir die studie. So ‘n breë basis stel die studie in staat om te verstaan hoe die ideologie oor meer as twee duisend jaar aangepas is. Die breë basis stel die studie ook in staat om 'n verskeidenheid verskillende gebare wat uitgebeeld word, te ondersoek. Hierdeur verskaf dit die leser met 'n beter begrip waarom sekere gebare gebruik is en hoe die onderliggende ideologie deur middel van hierdie bewegings gekommunikeer is. Die studie kom tot die gevolgtrekking dat terwyl die gebare 'n lewensgetroue voorkoms aan die voorstelling gee, hulle nie uitsluitlik onderliggende ideologiese boodskappe uitbeeld nie. Inteendeel, hulle versterk die reeds onderliggende ideologiese boodskap.