Augmented Reality (AR) has the possibility of interacting with virtual objects and real objects at the same time since it combines the real world with computer-generated contents seamlessly. However, most AR interface research uses general Virtual Reality (VR) interaction techniques without modification. In this research we develop a multimodal interface (MMI) for AR with speech and 3D hand gesture input. We develop a multimodal signal fusion architecture based on the user behaviour while interacting with the MMI that provides more effective and natural multimodal signal fusion. Speech and 3D vision-based free hand gestures are used as multimodal input channels. There were two user observations (1) a Wizard of Oz study and (2)Gesture modelling. With the Wizard of Oz study, we observed user behaviours of interaction with our MMI. Gesture modelling was undertaken to explore whether different types of gestures can be described by pattern curves. Based on the experimental observations, we designed our own multimodal fusion architecture and developed an MMI. User evaluations have been conducted to evaluate the usability of our MMI. As a result, we found that MMI is more efficient and users are more satisfied with it when compared to the unimodal interfaces. We also describe design guidelines which were derived from our findings through the user studies.

No description available.

This thesis studies the characteristics of the acoustic signal in speech, especially in speech directed to infants and in infant vocal development, to gain insight on essential aspects of speech processing, speech production and communicative interaction in early language acquisition. Three sets of experimental studies are presented in this thesis. From a phonetic point of view they investigate the fundamental processes involved in first language acquisition. The first set (study 1.1 and study 1.2) investigated how linguistic structure in the speech signal can be derived and which strategy infants and adults use to process information depending on its presentation. The second set (study 2.1 and study 2.2) studied acoustic consequences of the anatomical geometry of the infant vocal tract and the development of sensory-motor control for articulatory strategies. The third set of studies (study 3.1 and study 3.2) explored the infant's interaction with the linguistic environment, specifically how vocal imitation and reinforcement may assist infants to converge towards adult-like speech. The first set of studies suggests that structure and quality of simultaneous sensory input impact on the establishment of initial linguistic representations. The second set indicates that the anatomy of the infant vocal tract does not constrain the production of adult-like speech sounds and that some degree of articulatory motor control is present from six months of age. The third set of studies suggests that the adult interprets and reinforces vocalizations produced by the infant in a developmentally-adjusted fashion that can guide the infant towards the sounds of the ambient language. The results are discussed in terms of essential aspects of early speech processing and speech production that can be accounted for by biological general purpose mechanisms in the language learning infant. / För att köpa boken skicka en beställning till exp@ling.su.se/ To order the book send an e-mail to exp@ling.su.se