Zur Erledigung einer Aufgabe kann der Benutzer mit verschiedenen Endgeräten interagieren, welche unterschiedliche Interaktionsarten (Modalitäten) anbieten. Dabei gibt es jedoch kein Endgerät, welches alle erdenkbaren Modalitäten unterstützt. Aus diesem Grund wird eine komponentenbasierte Integrationsschicht auf Basis einer Komponentenplattform entwickelt, die dem Nutzer die gewünschte Freiheit bei der Wahl der Endgeräte und somit der Modalitäten ermöglicht. Als Ausgangsbasis dafür dient das W3C Multimodal Interaction Framework. Mit Hilfe der Integrationsschicht kann der Nutzer beispielsweise Endgeräteföderationen erzeugen, die einzeln oder gemeinsam zur Interaktion verwendet werden können. Die Integrationsschicht besitzt verschiedene Konzepte, um z.B. Geschäftslogik zur Laufzeit zu verteilen, Komponentenausfälle zu behandeln und die auf verschiedene Endgeräte verteilte Nutzerschnittstelle zu synchronisieren. Die entwickelten Konzepte wurden prototypisch implementiert, validiert und auf Performanz untersucht.

This thesis was motivated by a study of how robots can be taught by humans, with an emphasis on allowing persons without programming skills to teach robots. The focus of this thesis was to investigate what criteria could or should be used by a robot to evaluate whether a human teacher is (or potentially could be) a good teacher in robot learning by demonstration. In effect, choosing the teacher that can maximize the benefit to the robot using learning by imitation/demonstration. The study approached this topic by taking a technology snapshot in time to see if a representative example of research laboratory robot technology is capable of assessing teaching quality. With this snapshot, this study evaluated how humans observe teaching quality to attempt to establish measurement metrics that can be transferred as rules or algorithms that are beneficial from a robot's point of view. To evaluate teaching quality, the study looked at the teacher-student relationship from a human-human interaction perspective. Two factors were considered important in defining a good teacher: engagement and immediacy. The study gathered more literature reviews relating to further detailed elements of engagement and immediacy. The study also tried to link physical effort as a possible metric that could be used to measure the level of engagement of the teachers. An investigatory experiment was conducted to evaluate which modality the participants prefer to employ in teaching a robot if the robot can be taught using voice, gesture demonstration, or physical manipulation. The findings from this experiment suggested that the participants appeared to have no preference in terms of human effort for completing the task. However, there was a significant difference in human enjoyment preferences of input modality and a marginal difference in the robot's perceived ability to imitate. A main experiment was conducted to study the detailed elements that might be used by a robot in identifying a 'good' teacher. The main experiment was conducted in two subexperiments. The first part recorded the teacher's activities and the second part analysed how humans evaluate the perception of engagement when assessing another human teaching a robot. The results from the main experiment suggested that in human teaching of a robot (human-robot interaction), humans (the evaluators) also look for some immediacy cues that happen in human-human interaction for evaluating the engagement.

Face-to-face human communication is a multimodal and incremental process. Humans employ  different information channels (modalities) for their communication. Since some of these modalities are more error-prone to specic type of data, a multimodal communication can benefit from strengths of each modality and therefore reduce ambiguities during the interaction. Such interfaces can be applied to intelligent robots who operate in close relation with humans. With this approach, robots can communicate with their human colleagues in the same way they communicate with each other, thus leading to an easier and more robust human-robot interaction (HRI).In this work we suggest a new method for implementing multimodal interfaces in HRI domain and present the method employed on an industrial robot. We show that operating the system is made easier by using this interface. / Robot Colleague

Abstract Usability methodology has matured into a well-defined, industrially relevant field with its own findings, theories, and tools, with roots in applying information technology to user interfaces ranging from control rooms to computers, and more recently to mobile communications devices. The purpose is regularly to find out the users’ goals and to test whether a design fulfils the usability criteria. Properly applied, usability methods provide reliable and repeatable results, and are excellent tools in fine-tuning existing solutions The challenges of usability methodologies are in finding new concepts and predicting their characteristics before testing, especially when it comes to the relatively young field of mobile user interfaces. Current usability methods concentrate on utilising available user-interface technologies. They do not provide means to clearly identify, e.g., the potential of auditory or haptic output, or gestural input. Consequently, these new interaction techniques are rarely used, and the long-envisioned useful multimodal user interfaces are yet to appear, despite their assumed and existing potential in mobile devices. Even the most advocated and well-known multimodal interaction concepts, such as combined manual pointing and natural language processing, have not materialised in applications. An apparent problem is the lack of a way to utilise a usage environment analysis in finding out user requirements that could be met with multimodal user interfaces. To harness the full potential of multimodality, tools to identify both little or unused and overloaded modalities in current usage contexts are needed. Such tools would also help in putting possibly existing novel interaction paradigms in context and pointing out possible deficiencies in them. In this thesis, a novel framework for analysing the usage environment from a user-centric perspective is presented. Based on the findings, a designer can decide between offering a set of multiple devices utilising various user-interface modalities, or a single device that offers relevant modalities, perhaps by adapting to the usage environment. Furthermore, new techniques for creating mobile user interfaces utilising various modalities are proposed. The framework has evolved from the experiences gathered from the designs of experimental and actual product-level uni- and multimodal user interface solutions for mobile devices. It has generated novel multimodal interaction and interface techniques that can be used as building blocks in system designs.

The present dissertation concerns social organization of feedback in ongoing hairdressing education. The central aim is to explore feedback between teacher and student in multimodal interaction within classroom assessment, as co-production of action and student’s participation. Classroom assessment and feedback are understood as social actions situated in interaction. The empirical data consists of video recordings from two vocational schools. From 31 hours of video material, selections of feedback interactions were made. At first, teacher and student communication in feedback cycles and loops was analysed from a social semiotic perspective. Secondly, examples of student initiated feedback loops were analysed from a conversation analytic perspective. Thirdly, a single case of a teacher and a student interacting through feedback related to creative subject content was analysed from a conversation analytic perspective.  The analyses show the importance of collaborative use of artefacts and embodied communication in the production of mutual understanding; opening for student initiatives in actions of assessment as well as feedback. Silence and body position were found to be important resources giving the student space to display concern. Participation in feedback practices within creative subject content emerged in a trajectory of problem detection to problem solving, resulting in tacit dimensions of hairdressers’ knowing made explicit. The findings indicate the importance of taking a participatory perspective on multimodal interaction when exploring actions of assessment and feedback between teacher and student. This study shows how feedback is not only given from the teacher, but also locally produced as a collaborative practice between teachers and students, displaying tacit dimensions of professional knowledge and subject content. / Den här sammanläggningsavhandlingen handlar om återkoppling mellan lärare och elever i frisörutbildning. Med hjälp av videoinspelningar under pågående undervisning utforskas klassrumsbaserad bedömning och återkopplingspraktiker. Eleverna är dels nybörjare som arbetar på övningsdockor, dels elever i slutet av sin utbildning som tar emot kunder i sitt klassrum. Videoanalyserna ger möjlighet att få syn på multimodala aspekter av interaktionen och resultatet visar tydligt hur både kropp och material är viktiga resurser i deltagarnas samspel. Avhandlingen visar genomgående hur bedömning och återkoppling utgår både från lärares professionella yrkeskunnande och från elevernas egna initiativ på specifika områden där något problem behöver redas ut i relation till deras pågående arbeten. Studiernas detaljerade analyser av deltagarnas multimodala interaktion synliggör hur bedömning och återkoppling kan förstås som sociala handlingar där tal, kropp och artefakter samordnas för att skapa gemensam förståelse och mening. Avhandlingens bidrag riktar sig mot yrkesutbildning men även andra former för utbildning, eftersom bedömning och återkoppling praktiseras i varje klassrum och utbildningskontext. Avhandlingens delstudier ökar kunskapen om hur återkoppling kan förstås som sociala handlingar där varje deltagare bidrar i skapandet av ömsesidig respons. Att förstå interaktion som multimodal, synliggör hur återkoppling i pågående undervisning möjliggörs genom deltagande och samspel mellan lärare och elever.

This thesis explores interactive translation dictation (ITD), a translation technique that involves interaction with multimodal interfaces equipped with voice recognition (VR) technology throughout the entire translation process. Its main objective is to provide a solid theoretical background and an analysis of empirical qualitative and quantitative data that demonstrate ITD’s advantages and challenges, with a view to integrating this technique into the translation profession. Many empirical studies in human-computer interaction have strived to demonstrate the efficiency of voice input versus keyboard input. Although it was implicit in the earliest works that voice input was expected to completely replace—rather than complement—text-input devices, it was soon proposed that VR often performed better in combination with other input modes. This study introduces multimodal interaction to translation, taking advantage of the unparallelled robustness of commercially available voice-and-touch-enabled multimodal interfaces such as touch-screen computers and tablets. To that end, an experiment was carried out with 14 professional English-to-French translators, who performed a translation task either with the physical prototype of an ITD environment, or with a traditional keyboard-and-mouse environment. The hypothesis was that the prototypical environment would consistently provide translators with a better translator experience (TX) than the traditional environment, considering the translation process as a whole. The notion of TX as introduced in this work is defined as a translator’s perceptions of and responses to the use or anticipated use of a product, system or service. Both quantitative and qualitative data were collected using different methods, such as video and screen recording, input logging and semi-structured interviews. The combined analysis of objective and subjective usability measures suggests a better TX with the experimental environment versus the keyboard-and-mouse workstation, but significant challenges still need to be overcome for ITD to be fully integrated into the profession. Thus, this doctoral study provides a basis for better-grounded research in translator-computer interaction and translator-information interaction and, more specifically, for the design and development of an ITD environment, which is expected to support professional translators’ cognitive functions, performance and well-being. Lastly, this research aims to demonstrate that translation studies research, and translation technology in particular, needs to be more considerate of the translator, the TX, and the changing realities of the interaction between humans, computers and information in the twenty-first century.

Indoor flight of unmanned aerial vehicles (UAVs) has many applications in environments in which it is undesirable or dangerous for humans to be, such as military reconnaissance or searching for trapped victims in a collapsed building. However, limited visual feedback makes it difficult to pilot UAVs in cluttered and enclosed spaces. Haptic feedback combined with visual feedback has shown to reduce the number of collisions of UAVs in indoor environments; however, it has increased the mental workload of the operator. This thesis investigates the potential of combining novel haptic and 3D audio feedback to provide additional information to operators of UAVs in order to improve performance and reduce workload. Many haptic feedback algorithms, such as Time to Impact (TTI)~cite{Brandt2009}, have been developed to help pilot UAVs. This thesis compares TTI with two new haptic feedback algorithms: Omni-Directional Dynamics Springs (ODDS) and Velocity Scaled Omni-Directional Dynamic Springs (VSODDS). These novel algorithms are based on the idea that dynamic springs are attached to the haptic controller in all directions. This thesis is unique by augmenting visual and haptic feedback with real-time 3D audio feedback. Continuous Directional Graded Threshold (CDGT) and Discrete Directional Graded Threshold (DDGT) are two novel algorithms that were developed to provide 3D audio warning cues to operators. To reduce sensory overload, these algorithms play a graded audio alert cue in the direction of velocity and when within a threshold distance of an obstacle. In order to measure operator workload, many researchers have used subjective measures, which suffer from subject bias, preconceptions, and ordering. Instead of using a subjective measure, experimental data is used to objectively measure operator workload using behavioral entropy, which works on the idea that humans work to reduce entropy by skilled behavior. QuadSim, a robust and versatile indoor quadrotor simulator, was developed as a test bed for visual, haptic, and 3D audio feedback. Using QuadSim, a human subject experiment was performed to determine the effectiveness of haptic and 3D audio feedback on operator performance and workload. The results of the study indicate that haptic feedback significantly reduced the number of collisions and collision length. Operator workload was decreased in the side-to-side direction by VSODDS but was adversely increased by TTI. Overall, VSODDS outperformed the other haptic algorithms. Unlike haptic feedback, audio feedback proved to be neither helpful nor harmful in improving performance or reducing workload.

Ce travail de thèse élabore une approche de l’œuvre d’art comme accomplissement pratique, à partir d’un travail de terrain conduit lors de la préparation d’une exposition dans un centre d’art contemporain. Cette recherche, qui relève de l’analyse conversationnelle et de l’ethnométhodologie, vise ainsi à exploiter les acquis méthodologiques, théoriques et analytiques de ces approches pour la description d’un terrain artistique. Aborder l’œuvre « en pratiques », par l’analyse de situations d’interaction avec et autour des œuvres, permet de poser à nouveaux frais un ensemble de questions ayant trait à la définition de l’objet « œuvre d’art » et à la compréhension des activités artistiques et esthétiques. L’analyse détaillée de micro-activités accomplies par les professionnels de l’art (entrer en interaction dans l’espace d’une œuvre, donner des instructions pendant l’installation d’une œuvre, évaluer l’état d’un accrochage, introduire successivement différentes œuvres pendant une visite guidée) nous permet d’appréhender la constitution de l’œuvre comme un phénomène empirique, observé depuis l’écologie de l’exposition et les interactions entre ses acteurs. La thèse défend ainsi une approche descriptive des pratiques sociales situées qui constituent les œuvres. En cela, ce travail offre l’opportunité de s’interroger d’une part sur les manières dont un travail empirique relevant de la linguistique et de la sociologie peut contribuer à une réflexion esthétique, et d’autre part sur les formes de collaboration interdisciplinaire alors engagées par une recherche qui prend l’art pour domaine d’enquête. / This PhD dissertation develops an approach to artworks defined as practical accomplishments, drawing on fieldwork observation of the preparation of an exhibition in a center for contemporary art. This study draws on the analytical and theoretical findings of conversation analysis and ethnomethodology to investigate artistic and aesthetic activities. Approaching artworks as practical accomplishments, by analyzing interactions with and around artworks, contributes to redefining artworks as well as artistic and aesthetic activities. The detailed analysis of micro-activities accomplished by art professionals (entering the artwork’s space, giving instructions for installing the artwork, assessing the artwork installation, successively introducing different artworks in the course of a guided tour) allows for the description of the artwork’s constitution as an empirical phenomenon, as can be observed in the exhibition’s ecology and through the interactions between its actors. The study advocates a descriptive approach to social and situated practices which produce artworks. It endeavors to offer an empirical contribution, from a linguistic and sociological perspective, to theoretical debates in aesthetics and, incidentally, to expand the study of art as an interdisciplinary field of research.

Zur Erledigung einer Aufgabe kann der Benutzer mit verschiedenen Endgeräten interagieren, welche unterschiedliche Interaktionsarten (Modalitäten) anbieten. Dabei gibt es jedoch kein Endgerät, welches alle erdenkbaren Modalitäten unterstützt. Aus diesem Grund wird eine komponentenbasierte Integrationsschicht auf Basis einer Komponentenplattform entwickelt, die dem Nutzer die gewünschte Freiheit bei der Wahl der Endgeräte und somit der Modalitäten ermöglicht. Als Ausgangsbasis dafür dient das W3C Multimodal Interaction Framework. Mit Hilfe der Integrationsschicht kann der Nutzer beispielsweise Endgeräteföderationen erzeugen, die einzeln oder gemeinsam zur Interaktion verwendet werden können. Die Integrationsschicht besitzt verschiedene Konzepte, um z.B. Geschäftslogik zur Laufzeit zu verteilen, Komponentenausfälle zu behandeln und die auf verschiedene Endgeräte verteilte Nutzerschnittstelle zu synchronisieren. Die entwickelten Konzepte wurden prototypisch implementiert, validiert und auf Performanz untersucht.

Les entrées et sorties unimodales dans les systèmes actuels ont atteint une maturité reconnue, avec les applications tactiles ou par les services distribués pour la géo-localisation ou la reconnaissance de la parole, du son ou l’'image. Cependant, l'intégration et l’instanciation de toutes ces modalités, manque d’une gestion intelligente du contexte d’acquisition et de restitution basée sur des notions fortement formalisées mais reflétant le sens commun. Ceci demande un comportement plus dynamique du système avec une approche plus adéquate pour gérer l'environnement de l'utilisateur. Cependant,la technologie nécessaire pour atteindre un tel objectif n’est pas encore disponible de façon standardisée, tant au niveau des descriptions fonctionnelles des services unimodaux que de leur description sémantique. Ceci est aussi le cas pour les architectures multimodales, où la composante sémantique est produite par chaque projet sans un accord commun pour assurer l’interoperabilité et est souvent limitée au traitement des entrées et sorties ou aux processus de fusion/fission strictement nécessaires au projet.Pour combler cette lacune, nous proposons une approche sémantique orientée services pour une architecture multimodale générique qui vise à améliorer la description et la découverte des composants de modalité pour les services d'assistance: l'architecture SOA2m. Cette architecture se veut entièrement focalisée sur la multimodalité et enrichie avec des technologies sémantiques car nous croyons que cette orientation permettra d'enrichir le comportement autonome des applications multimodales, avoir une perception robuste des échanges, et contrôler l'intégration sémantique. / Unimodal inputs and outputs in current systems have become very mature with touch applications or distributed services for geo-localization or speech, audio and image recognition. However, the integration and instantiation of all these modalities, lack of an intelligent management of the acquisition and restitution context, based on highly formalized notions reflecting common sense. This requires a more dynamic behavior of the system with a more appropriate approach to manage the user environment.However, the technology required to achieve such a goal is not yet available in a standardized manner, both in terms of the functional description of unimodal services and in terms of their semantic description. This is also the case for multimodal architectures, where the semantic management is produced by each project without a common agreement in the field to ensure inter-operability, and it is often limited to the processing of inputs and outputs or fusion / fission mechanisms. To fill this gap, we propose a semantic service-oriented generic architecture for multimodal systems. This proposal aims to improve the description and the discovery of modality components for assistance services: this is the architecture SOA2m. This architecture is fully focused on multimodality and it is enriched with semantic technologies because we believe that this approach will enhance the autonomous behavior of multimodal applications, provide a robust perception of the user-system exchanges, and help in the control of the semantic integration of the human-computer interaction.As a result, the challenge of discovery is addressed using the tools provided by the field of the semantic web services