Paper-based user manuals that provide assembly and disassembly instructions often do so with a combination of diagrams supported with textual information that clarifies how to perform the tasks. Mobile devices are emerging as a multimedia platform for providing on-demand training due to their portability. Mobile devices have limited screen size; as a result, the text instructions associated with the diagrams can produce clutter and occlusion on the screen. Also, too much information if fed through a single sensory channel (visual) may result in excessive cognitive load on the working memory of the human brain, thus hindering the learning process. In this work, two user studies were conducted to investigate the tradeoffs of using text, voice, and a combination of both modalities on the learning experience in a just-in-time mobile learning scenario. In such a scenario end-users are managing two very visual tasks at the same time; i.e., the primary task of carrying out the assembly/disassembly job and the secondary task of learning how to perform the task.

No description available.

Although robotics is well established as a research field, there has been relatively little work on human-robot collaboration. This type of collaboration is going to become an increasingly important issue as robots work ever more closely with humans. Clearly, there is a growing need for research on human-robot collaboration and communication between humans and robotic systems. Research into human-human communication can be used as a starting point in developing a robust human-robot collaboration system. Previous research into collaborative efforts with humans has shown that grounding, situational awareness, a common frame of reference and spatial referencing are vital in effective communication. Therefore, these items comprise a list of required attributes of an effective human-robot collaborative system. Augmented Reality (AR) is a technology for overlaying three-dimensional virtual graphics onto the user's view of the real world. It also allows for real time interaction with these virtual graphics, enabling a user to reach into the augmented world and manipulate it directly. The internal state of a robot and its intended actions can be displayed through the virtual imagery in the AR environment. Therefore, AR can bridge the divide between human and robotic systems and enable effective human-robot collaboration. This thesis describes the work involved in developing the Augmented Reality Human-Robot Collaboration (AR-HRC) System. It first garners design criteria for the system from a review of communication and collaboration in human-human interaction, the current state of Human-Robot Interaction (HRI) and related work in AR. A review of research in multimodal interfaces is then provided highlighting the benefits of using such an interface design. Therefore, an AR multimodal interface was developed to determine if this type of design improved performance over a single modality design. Indeed, the multimodal interface was found to improve performance, thereby providing the impetus to use a multimodal design approach for the AR-HRC system. The architectural design of the system is then presented. A user study conducted to determine what kind of interaction people would use when collaborating with a mobile robot is discussed and then the integration of a mobile robot is described. Finally, an evaluation of the AR-HRC system is presented.

Augmented Reality (AR) has the possibility of interacting with virtual objects and real objects at the same time since it combines the real world with computer-generated contents seamlessly. However, most AR interface research uses general Virtual Reality (VR) interaction techniques without modification. In this research we develop a multimodal interface (MMI) for AR with speech and 3D hand gesture input. We develop a multimodal signal fusion architecture based on the user behaviour while interacting with the MMI that provides more effective and natural multimodal signal fusion. Speech and 3D vision-based free hand gestures are used as multimodal input channels. There were two user observations (1) a Wizard of Oz study and (2)Gesture modelling. With the Wizard of Oz study, we observed user behaviours of interaction with our MMI. Gesture modelling was undertaken to explore whether different types of gestures can be described by pattern curves. Based on the experimental observations, we designed our own multimodal fusion architecture and developed an MMI. User evaluations have been conducted to evaluate the usability of our MMI. As a result, we found that MMI is more efficient and users are more satisfied with it when compared to the unimodal interfaces. We also describe design guidelines which were derived from our findings through the user studies.

Intelligent devices are quickly becoming necessities to support our activities during both work and play. We are already bound in a symbiotic relationship with these devices. An unfortunate effect of the pervasiveness of intelligent devices is the substantial investment of our time and effort to communicate intent. Even though our increasing reliance on these intelligent devices is inevitable, the limits of conventional methods for devices to perceive human expression hinders communication efficiency. These constraints restrict the usefulness of intelligent devices to support our activities. Our communication time and effort must be minimized to leverage the benefits of intelligent devices and seamlessly integrate them into society. Minimizing the time and effort needed to communicate our intent will allow us to concentrate on tasks in which we excel, including creative thought and problem solving. <br /><br /> An intuitive method to minimize human communication effort with intelligent devices is to take advantage of our existing interpersonal communication experience. Recent advances in speech, hand gesture, and facial expression recognition provide alternate viable modes of communication that are more natural than conventional tactile interfaces. Use of natural human communication eliminates the need to adapt and invest time and effort using less intuitive techniques required for traditional keyboard and mouse based interfaces. <br /><br /> Although the state of the art in natural but isolated modes of communication achieves impressive results, significant hurdles must be conquered before communication with devices in our daily lives will feel natural and effortless. Research has shown that combining information between multiple noise-prone modalities improves accuracy. Leveraging this complementary and redundant content will improve communication robustness and relax current unimodal limitations. <br /><br /> This research presents and evaluates a novel multimodal framework to help reduce the total human effort and time required to communicate with intelligent devices. This reduction is realized by determining human intent using a knowledge-based architecture that combines and leverages conflicting information available across multiple natural communication modes and modalities. The effectiveness of this approach is demonstrated using dynamic hand gestures and simple facial expressions characterizing basic emotions. It is important to note that the framework is not restricted to these two forms of communication. The framework presented in this research provides the flexibility necessary to include additional or alternate modalities and channels of information in future research, including improving the robustness of speech understanding. <br /><br /> The primary contributions of this research include the leveraging of conflicts in a closed-loop multimodal framework, explicit use of uncertainty in knowledge representation and reasoning across multiple modalities, and a flexible approach for leveraging domain specific knowledge to help understand multimodal human expression. Experiments using a manually defined knowledge base demonstrate an improved average accuracy of individual concepts and an improved average accuracy of overall intents when leveraging conflicts as compared to an open-loop approach.

Intelligent devices are quickly becoming necessities to support our activities during both work and play. We are already bound in a symbiotic relationship with these devices. An unfortunate effect of the pervasiveness of intelligent devices is the substantial investment of our time and effort to communicate intent. Even though our increasing reliance on these intelligent devices is inevitable, the limits of conventional methods for devices to perceive human expression hinders communication efficiency. These constraints restrict the usefulness of intelligent devices to support our activities. Our communication time and effort must be minimized to leverage the benefits of intelligent devices and seamlessly integrate them into society. Minimizing the time and effort needed to communicate our intent will allow us to concentrate on tasks in which we excel, including creative thought and problem solving. <br /><br /> An intuitive method to minimize human communication effort with intelligent devices is to take advantage of our existing interpersonal communication experience. Recent advances in speech, hand gesture, and facial expression recognition provide alternate viable modes of communication that are more natural than conventional tactile interfaces. Use of natural human communication eliminates the need to adapt and invest time and effort using less intuitive techniques required for traditional keyboard and mouse based interfaces. <br /><br /> Although the state of the art in natural but isolated modes of communication achieves impressive results, significant hurdles must be conquered before communication with devices in our daily lives will feel natural and effortless. Research has shown that combining information between multiple noise-prone modalities improves accuracy. Leveraging this complementary and redundant content will improve communication robustness and relax current unimodal limitations. <br /><br /> This research presents and evaluates a novel multimodal framework to help reduce the total human effort and time required to communicate with intelligent devices. This reduction is realized by determining human intent using a knowledge-based architecture that combines and leverages conflicting information available across multiple natural communication modes and modalities. The effectiveness of this approach is demonstrated using dynamic hand gestures and simple facial expressions characterizing basic emotions. It is important to note that the framework is not restricted to these two forms of communication. The framework presented in this research provides the flexibility necessary to include additional or alternate modalities and channels of information in future research, including improving the robustness of speech understanding. <br /><br /> The primary contributions of this research include the leveraging of conflicts in a closed-loop multimodal framework, explicit use of uncertainty in knowledge representation and reasoning across multiple modalities, and a flexible approach for leveraging domain specific knowledge to help understand multimodal human expression. Experiments using a manually defined knowledge base demonstrate an improved average accuracy of individual concepts and an improved average accuracy of overall intents when leveraging conflicts as compared to an open-loop approach.

The paper deals with the creation of interface based on multiple input signals, i.e. multimodal interface. For this purpose analyzes the benefits of the approach to communicate with the device that way. The work also includes an overview of the level at which you can perform data fusion, and different approaches to the layout of the system architecture for multimodal data processing. The important part is the actual design of the system, where for the interface was chosen distributed architecture using software agents for processing inputs. As a method for data integration was picked hybrid fusion based on dialog driven and unification strategy. The result should be an interface for media center control and interaction with other devices around the user.

Les troubles neuropsychologiques sont très répandus et posent de réels problèmes de santé publique. En particulier, dans notre société moderne où la communication écrite est omniprésente, la dyslexie peut s’avérer excessivement handicapante. On remarque néanmoins que le diagnostic et la remédiation de cette pathologie restent délicats et manquent d’uniformisation. Ceci semble malheureusement inhérent à la caractérisation clinique par exclusion de la dyslexie, à la multitude de praticiens différents impliqués dans une telle prise en charge ainsi qu’au manque d’objectivité de certaines méthodes existantes. A ce titre, nous avons décidé d’investiguer les possibilités offertes par l’informatique actuelle pour surmonter ces barrières. Effectivement, nous avons supposé que la démocratisation des systèmes informatiques et leur puissance de calcul pourraient en faire un outil de choix pour pallier les difficultés rencontrées lors de la prise en charge de la dyslexie. Cette recherche nous a ainsi mené à étudier les techniques, aussi bien logicielles que matérielles, pouvant conduire au développement d’un système bon marché et évolutif qui serait capable d’assister un changement bénéfique et progressif des pratiques qui entourent cette pathologie. Avec ce projet, nous nous plaçons définitivement dans un courant innovant au service de la qualité des soins et des aides apportées aux personnes souffrant d’un handicap. Notre travail a ainsi consisté à identifier différents axes d’amélioration que l’utilisation de l’outil informatique rend possible. Chacun de ces axes a alors pu faire l’objet de recherches exhaustives, de modélisations et de développements de prototypes. Nous avons également réfléchi à la méthodologie à mettre en œuvre pour concevoir un tel système dans sa globalité. En particulier, nos réflexions et ces différents accomplissements nous ont permis de définir un framework logiciel propice à l’implémentation d’une plate-forme logicielle que nous avons appelée la PAMMA. Cette plate-forme devrait théoriquement pouvoir disposer de tous les outils permettant le développement souple et efficace d’applications médicales intégrant des processus métiers. Il est ainsi attendu de ce système qu’il permette le développement d’applications, pour la prise en charges des patients dyslexiques, conduisant à un diagnostic plus rapide et plus précis ainsi qu’à une remédiation plus adaptée et plus efficace. De notre effort d’innovation ressortent des perspectives encourageantes. Cependant, ce type d’initiative ne peut se concrétiser qu’autour de collaborations pluridisciplinaires disposant de nombreux moyens fonctionnels, techniques et financiers. La constitution d’un tel consortium semble donc être la prochaine étape nécessaire à l’obtention des financements pour réaliser un premier prototype fonctionnel de la PAMMA, ainsi que de premières applications. Des études cliniques pourront être alors menées pour prouver indubitablement l’efficacité d’une telle approche dans le cadre de la prise en charge de la dyslexie, ainsi qu’éventuellement d’autres troubles neuropsychologiques. / Neuropsychological disorders are widespread and generate real public health problems. In particular in our modern society, where written communication is ubiquitous, dyslexia can be extremely disabling. Nevertheless we can note that the diagnosis and remediation of this pathology are fastidious and lack of standardization. Unfortunately it seems inherent to the clinical characterization of dyslexia by exclusion, to the multitude of different practitioners involved in such treatment and to the lack of objectivity of some existing methods. In this respect, we decided to investigate the possibilities offered by modern computing to overcome these barriers. Indeed we have assumed that the democratization of computer systems and their computing power could make of them a perfect tool to alleviate the difficulties encountered in the treatment of dyslexia. This research has led us to study the techniques software as well as hardware, which can conduct to the development of an inexpensive and scalable system able to attend a beneficial and progressive changing of practices in this pathology field. With this project we put ourselves definitely in an innovative stream serving quality of care and aid provided to people with disabilities. Our work has been identifying different improvement areas that the use of computers enables. Then each of these areas could then be the subject of extensive research, modeling and prototype developments. We also considered the methodology for designing this kind of system as a whole. In particular our thoughts and these accomplishments have allowed us to define a software framework suitable for implementing a software platform that we called the PAMMA. This platform should theoretically have access to all the tools required for the flexible and efficient development of medical applications integrating business processes. In this way it is expected that this system allows the development of applications for caring dyslexic patients thus leading to a faster and more accurate diagnosis and a more appropriate and effective remediation. Of our innovation efforts emerge encouraging perspectives. However such initiatives can only be achieved within multidisciplinary collaborations with many functional, technical and financial means. Creating such a consortium seems to be the next required step to get a funding necessary for realizing a first functional prototype of the PAMMA, as well as its first applications. Some clinical studies may be conducted to prove undoubtedly the effectiveness of such an approach for treating dyslexia and eventually other neuropsychological disorders.