L'automatisation de la conduite est un processus en cours qui est en train de changer radicalement la façon dont les gens voyagent et passent du temps dans leur voiture pendant leurs déplacements. Les véhicules conditionnellement automatisés libèrent les conducteurs humains de la surveillance et de la supervision du système et de l'environnement de conduite, leur permettant d'effectuer des activités secondaires pendant la conduite, mais requièrent qu’ils puissent reprendre la tâche de conduite si nécessaire. Pour les conducteurs, il est essentiel de comprendre les capacités et les limites du système, d’en reconnaître les notifications et d'interagir de manière adéquate avec le véhicule pour assurer leur propre sécurité et celle des autres usagers de la route. À cause de la diversité des situations de conduite que le conducteur peut rencontrer, les programmes traditionnels de formation peuvent ne pas être suffisants pour assurer une compréhension efficace de l'interaction entre le conducteur humain et le véhicule pendant les transitions de contrôle. Il est donc nécessaire de permettre aux conducteurs de vivre ces situations avant leur première utilisation du véhicule. Dans ce contexte, la Réalité Mixte constitue un outil d'apprentissage et d'évaluation des compétences potentiellement efficace qui permettrait aux conducteurs de se familiariser avec le véhicule automatisé et d'interagir avec le nouvel équipement dans un environnement sans risque. Si jusqu'à il y a quelques années, les plates-formes de Réalité Mixte étaient destinées à un public de niche, la démocratisation et la diffusion à grande échelle des dispositifs immersifs ont rendu leur adoption plus accessible en termes de coût, de facilité de mise en œuvre et de configuration. L'objectif de cette thèse est d'étudier le rôle de la réalité mixte dans l'acquisition de compétences pour l'interaction d'un conducteur avec un véhicule conditionnellement automatisé. En particulier, nous avons exploré le rôle de l'immersion dans le continuum de la réalité mixte en étudiant différentes combinaisons d'espaces de visualisation et de manipulation et la correspondance entre le monde virtuel et le monde réel. Du fait des contraintes industrielles, nous avons limité les candidats possibles à des systèmes légers portables, peu chers et facilement accessibles; et avons analysé l’impact des incohérences sensorimotrices que ces systèmes peuvent provoquer sur la réalisation des activités dans l’environnement virtuel. À partir de ces analyses, nous avons conçu un programme de formation visant l'acquisition des compétences, des règles et des connaissances nécessaires à l'utilisation d'un véhicule conditionnellement automatisé. Nous avons proposé des scénarios routiers simulés de plus en plus complexes pour permettre aux apprenants d’interagir avec ce type de véhicules dans différentes situations de conduite. Des études expérimentales ont été menées afin de déterminer l'impact de l'immersion sur l'apprentissage, la pertinence du programme de formation conçu et, à plus grande échelle, de valider l'efficacité de l'ensemble des plateformes de formation par des mesures subjectives et objectives. Le transfert de compétences de l'environnement de formation à la situation réelle a été évalué par des essais sur simulateurs de conduite haut de gamme et sur des véhicules réels sur la voie publique. / Driving automation is an ongoing process that is radically changing how people travel and spend time in their cars during journeys. Conditionally automated vehicles free human drivers from the monitoring and supervision of the system and driving environment, allowing them to perform secondary activities during automated driving, but requiring them to resume the driving task if necessary. For the drivers, understanding the system’s capabilities and limits, recognizing the system’s notifications, and interacting with the vehicle in the appropriate way is crucial to ensuring their own safety and that of other road users. Because of the variety of unfamiliar driving situations that the driver may encounter, traditional handover and training programs may not be sufficient to ensure an effective understanding of the interaction between the human driver and the vehicle during transitions of control. Thus, there is the need to let drivers experience these situations before their first ride. In this context, Mixed Reality provides potentially valuable learning and skill assessment tools which would allow drivers to familiarize themselves with the automated vehicle and interact with the novel equipment involved in a risk-free environment. If until a few years ago these platforms were destined to a niche audience, the democratization and the large-scale spread of immersive devices since then has made their adoption more accessible in terms of cost, ease of implementation, and setup. The objective of this thesis is to investigate the role of Mixed Reality in the acquisition of competences needed for a driver’s interaction with a conditionally automated vehicle. In particular, we explored the role of immersion along the Mixed Reality continuum by investigating different combinations of visualization and manipulation spaces and the correspondence between the virtual and the real world. For industrial constraints, we restricted the possible candidates to light systems that are portable, cost-effective and accessible; we thus analyzed the impact of the sensorimotor incoherences that these systems may cause on the execution of tasks in the virtual environment. Starting from these analyses, we designed a training program aimed at the acquisition of skills, rules and knowledge necessary to operate a conditionally automated vehicle. In addition, we proposed simulated road scenarios with increasing complexity to suggest what it feels like to be a driver at this level of automation in different driving situations. Experimental user studies were conducted in order to determine the impact of immersion on learning and the pertinence of the designed training program and, on a larger scale, to validate the effectiveness of the entire training platform with self-reported and objective measures. Furthermore, the transfer of skills from the training environment to the real situation was assessed with test drives using both high-end driving simulators and actual vehicles on public roads.

Recent progress in the overlay and registration of digital information on the users workspace in a spatially meaningful way has allowed mixed reality (MR) to become a more effective operational medium. In the area of medical surgery, surgeons are conveyed with information such as the incisions location, regions to be avoided, diseased tissues, etc, while staying in and keeping their original working environment. The main objective of this Thesis is identifying theoretical and practical basis for how mixed reality interfaces might provide support and augmentation maximizing the continuity of interaction. We start proposing a set of design principles organized in a design space which allows to identify continuity interaction properties at an early stage of the development system. Once the abstract design possibilities have been identified and a concrete design decision has been taken, an implementational strategy can be developed. Two approaches were investigated: markerless and marker-based. The last one is used to provide surgeons with guidance on an osteotomy task in the maxillo-facial surgery. The evaluation process applies usability tests with users to validate the augmented guidance in different scenarios and to study the influence of different design variables in the final user interaction. As a result we have found a model to describe the contribution factors of each variable for the continuity of the user interaction. We suggest that this methodology can be applied mainly to those applications in which smooth connections and interactions, with virtual and real environments, are critical for the system; i.e. surgery, drivers applications or pilot simulations.

This thesis aims to answer the following question : “How could gestures and facial expressions be used to control the behavior of an interactive entertaining application?”. An answer to this question is presented and illustrated in the context of mixed reality interactive storytelling. The first part focuses on the description of the Artificial Intelligence (AI) mechanisms that are used to model and control the behavior of the application. We present an efficient real-time hierarchical planning engine, and show how active modalities (such as intentional gestures) and passive modalities (such as facial expressions) can be integrated into the planning algorithm, in such a way that the narrative (driven by the behavior of the virtual characters inside the virtual world) can effectively evolve in accordance with user interactions. The second part is devoted to the automatic recognition of user interactions. After briefly describing the implementation of a simple but robust rule-based gesture recognition system, the emphasis is set on facial expression recognition. A complete solution integrating state-of-the-art techniques along with original contributions is drawn. It includes face detection, facial feature extraction and analysis. The proposed approach combines statistical learning and probabilistic reasoning in order to deal with the uncertainty associated with the process of modeling facial expressions.

Recent progress in the overlay and registration of digital information on the users workspace in a spatially meaningful way has allowed mixed reality (MR) to become a more effective operational medium. In the area of medical surgery, surgeons are conveyed with information such as the incisions location, regions to be avoided, diseased tissues, etc, while staying in and keeping their original working environment. The main objective of this Thesis is identifying theoretical and practical basis for how mixed reality interfaces might provide support and augmentation maximizing the continuity of interaction. We start proposing a set of design principles organized in a design space which allows to identify continuity interaction properties at an early stage of the development system. Once the abstract design possibilities have been identified and a concrete design decision has been taken, an implementational strategy can be developed. Two approaches were investigated: markerless and marker-based. The last one is used to provide surgeons with guidance on an osteotomy task in the maxillo-facial surgery. The evaluation process applies usability tests with users to validate the augmented guidance in different scenarios and to study the influence of different design variables in the final user interaction. As a result we have found a model to describe the contribution factors of each variable for the continuity of the user interaction. We suggest that this methodology can be applied mainly to those applications in which smooth connections and interactions, with virtual and real environments, are critical for the system; i.e. surgery, drivers applications or pilot simulations.

Det här examensarbetet på D-nivå har utförts i samarbete men företaget XM reality i Linköping. Företaget jobbar med att ta fram olika system inom området mixed reality, som på svenska översätts till förhöjd verklighet.   Syftet med projektet var att utforma två olika handhållna projektorsystem. Produkterna skulle utformas för att passa in i sjukhusmiljö. Båda enheterna har samma funktion men innehåller olika komponenter. Produkterna är anpassade till en ny teknik som företaget har utvecklat. Den nya tekniken är en mixed reality applikation som gör det möjligt att projicera skiktscanningar, CT eller MRI, direkt på patientens kropp. Målet med projektet var att göra två fungerade modeller. Modellerna skall fungera som prototyper i den medicinska forskarstudien som projektet ingår i.   En designstrategi utformades till företaget som skall uttrycka företagets image och vision. I arbetet med designstrategin togs ett antal kärnvärden fram. Kärnvärdena blev high tech, professionell, framtid, exklusiv, kvalité, hållbar samt kompakt vilket skall gälla för företagets samtliga produkter. Till produktsegmentet sjukhusprodukter tillades kärnvärdena ergonomisk samt dynamisk.   Arbetet med att utveckla produkterna började med litterära ergonomiska studier som följdes upp av fysiska användarstudier. Under arbetats gång har flera olika metoder tillämpats. Metoderna har används för att fatta beslut men även som stöd i den kreativa gestaltningsprocessen.   Resultatet är två ergonomiskt anpassade handhållna enheter. Enheterna är utformade för att passa in i sjukhusmiljön. De är smidiga samt enkla att använda. Båda produkterna är utformade efter designstrategin och för att förmedla företagets image. De fysiska modeller som gjordes innehåller fungerande komponenter som går att bytas ut. / This thesis D-level is done in collaboration with the company XM reality that is located in Linköping, Sweden. The company is working with the development of various systems in the field of mixed reality.   The task was to design two different hand-held projector systems. The products should be designed to fit in to the hospital environment. Both products have the same function but contain different components. The products are adapted to a new technology that the company has developed. The new technology is a mixed reality application that makes it possible to project CT or MRI scans directly on to the patient's body. The goal of the project was to make two functioning models. The models are to be used as prototypes in the medical research study the project is a part of.   A design strategy was created for the company to reflect the company’s image and vision. In the process of developing the design strategy, a number of core values were established. The core values were high-tech, professional, futuristic, exclusive, quality, durable and compact, which should to be valid for all the company’s products. To the product segment of hospital products were the core values ergonomic and dynamic added.   The work to develop the products began with literary ergonomic studies that were followed up by user studies. During the working process, several different methods were used. The methods were used not only to make decisions but also to help the creative design process.   The result is two ergonomically designed hand-held devices. The products are designed to fit into the hospital environment. They are slim and easy to use. Both products are designed along the design strategy and to communicate the company's image. The physical models that were made contain functional components that can be changed.

D. Tech. Electrical Engineering. / Developes a wheelchair motion platform whereby its user may be introduced into a simulated world. This simulated world is then required to be closely related to real world spaces that will be encountered by a disabled person using a wheelchair as a mobility aid. The wheelchair to be accommodated in the simulation environment may have multiple mechanical construct possibilities. The wheelchair used on the simulation platform needs to be driven by a combination of two wheels, as is generally found on manual and electric wheelchairs. The final objective was to design the simulation as closely as possible to the real world in order to use the VS-1 motion platform for architectural evaluations, possible training and general research in the field of simulators used in an enabled environment.

This research examines mixed reality meetings, a context where individuals attend to both face-to-face group members while multitasking with technology. In these meetings, members engage simultaneously with those physically present and those outside of the meeting (virtual communication partners). Technology multitasking in meetings has a dual effect: it not only impacts the individual user, it has the potential to transform how collocated groups communicate and work together since attention becomes fragmented across multiple competing tasks. Qualitative and quantitative methods were used to investigate mixed reality meetings across four themes: (1) the factors contributing to the likelihood to multitask based on meeting type, polychronicity (one’s preference for multitasking), and cohesion beliefs, (2) behavior during mixed reality assessed by copresence management, (3) attitudes toward technology multitasking, and (4) subjective outcomes measured by perceived productivity and meeting satisfaction. The qualitative data set consists of fieldwork from a global software company and interviews with 8 information workers. The quantitative data are comprised of survey results from the fieldwork site (n=156) and an online panel of information workers (n=110). Results indicate that information workers perceive distinct meeting types that are associated with implicit norms for appropriate technology multitasking. These norms varied based on the relevance of a meeting segment and if a power figure was present. A higher preference score for multitasking (high polychronicity) was significantly correlated with increased technology multitasking and perceived productivity. Members of cohesive teams exhibited the most technology multitasking and perceived their teammates multitasking as appropriate. However, outsiders who exhibited the same behaviors were viewed as rude and distracting. Overall, information workers who multitasked during meetings did so with electronic communication tasks (e-mail and instant messaging) as opposed to other computing tasks (e.g. writing documents, researching information). These findings are discussed in relation to psychological studies on multitasking, computer-supported cooperative work, and social constructionist views of technology use. This dissertation is a contribution to the assessment of technology use in social settings, particularly in organizations where tasks are often interrupted and a reliance on electronic communication tools impacts how people manage and accomplish work. / text

Zugl.: Saarbrücken, Univ., Diss., 2006

Techn. Universiẗat, Diss., 2006--Darmstadt.

A manutenção industrial é um dos grandes desafios na área de automação e manufatura da indústria. Este desafio tem se apresentado, em âmbito nacional, pela adoção da prática “predizer e prevenir” ao invés de “falhar e consertar”. Enquanto o Brasil caminha para uma mudança de paradigma de manutenção, o cenário internacional se volta para soluções em manufatura sustentável que atendam aos requisitos “eco and green”. E a manutenção é parte crucial deste novo contexto de produção. Entretanto, somente nos anos noventa, a indústria européia começa a delinear estratégias de manutenção. A complexidade e multidisciplinaridade deste tema tem sido o principal obstáculo para melhoras substanciais na disciplina de manutenção. São nas estratégias de manutenção, aliadas ao crescente avanço em Tecnologias da Informação e Comunicação (TIC), que aparecem as possibilidades de superar as limitações e os desafios desta área de pesquisa. A Manutenção Inteligente (MI) é uma dessas estratégias e utiliza o paradigma baseado em condição (Condition-based maintenance) para aplicação da técnica “predizer e prevenir”. Contudo, uma das principais barreiras na adoção de MI é a sua transferência efetiva para o operador em chão de fábrica. O conhecimento gerado pelos sistemas MI deve ser transformado em informações inteligíveis e úteis para o operador. Entretanto, o acesso, o entendimento e o uso destes dados, durante a atividade de manutenção, não é um processo trivial, devido à sobrecarga de informações e sistemas envolvidos nestes processos. Surge, então, como alternativa para contornar este problema, o uso de técnicas de realidade mista como meio de potencializar a interface homem-máquina. A escolha, neste trabalho, por interfaces mistas, objetiva, não apenas a visualização, mas a integração e o gerenciamento da informação. A fim de alcançar estes objetivos, é desenvolvida uma abordagem baseada em modelo. A primeira etapa consiste na definição de um modelo conceitual que integre os diferentes domínios envolvidos na solução proposta. Nesta fase, foram identificadas três áreas da manutenção em que a realidade mista pode permitir um ganho significativo: diagnóstico, planejamento e segurança. A segunda etapa consiste na descrição formal de um modelo de dados para integrar e gerenciar informações de diversos formatos. Finalmente, é implementado um sistema de visualização para validar o modelo de dados em um conjunto de estudos de caso da indústria. Objetiva-se, com esta abordagem: (i) facilitar a identificação de quais componentes apresentam problemas, o que auxilia no processo de tomada de decisão nas tarefas de manutenção; (ii) a inclusão do human in the loop, com o sistema MI, em que o operador poderá solicitar dados e medições adicionais durante a atividade de manutenção; (iii) fornecer um modelo que integre dados de formatos distintos, que viabilize uma maneira fácil e flexível de configurar o ambiente virtual. A visualização mista, utilizada pelo operador de manutenção em chão de fábrica, pode representar uma vantagem competitiva para indústria, além de prover uma maior flexibilidade e segurança para o operador durante a atividade de manutenção. / Industrial maintenance is one of the major challenges in industrial automation and manufacturing area. This challenge presents itself on the national context by the transformation to "predict and prevent" rather than "fail and fix ". While the Brazil moves toward a paradigm shift of maintenance, the international context searches manufacturing sustainable solutions that reply according to eco and green requirements and the maintenance is crucial part of this new production environment. However, it was only in the nineties that European industry started discusses maintenance strategies. The maintenance complexity and the multidisciplinary have been the main limitations for substantial improvements in the maintenance discipline. There are maintenance strategies coupled with the advances in Information and Communication Technologies (ICT) that arise the possibilities to overcome the limitations and challenges of this research area. The intelligent maintenance (IM) is one of these strategies and it uses the Condition-Based Maintenance paradigm (CBM). On the other hand one of the obstacles about the IM adoption is its effective transfer to the operator on the factory floor. The knowledge from IM systems must be transformed in understandable information and useful to the operator. However, the access, the understanding and the use of such information during the maintenance activity are not a trivial activity due the data and systems overload involved in these processes. Thus, arises as an alternative to solve this problem, the using mixed reality techniques to human-computer interface. The use of mixed interfaces in this thesis will aim not only to visualization but the integration and management of information. To achieve these goals was developed a model-based approach. The first step consisted in defining a conceptual model that address the domains involved in the solution proposed. At this stage were identified three main areas where the mixed reality could allow a significant change to maintenance activities: diagnosis, planning and security. The second step consisted in the formal description of a data model for integrating and management of information from multiple formats. Finally a visualization system was implemented to validate the data model in a set of case studies of industry. The goals of this thesis are: (i) to facilitate the identification of which components present problems in order to help in the decision making process of the maintenance tasks; (ii) the inclusion of the "human in the loop" into intelligent maintenance system, where the operator can request measurements and additional information during the maintenance process; (iii) to supply a model to integrate data of different formats making available an easy and flexible tool for configuring the virtual environment. The mixed reality, used by the maintenance operator on factory floor, may represent a competitive advantage for industry and provide greater flexibility and safety for operator during maintenance activity.