This thesis aims to answer the following question : “How could gestures and facial expressions be used to control the behavior of an interactive entertaining application?”. An answer to this question is presented and illustrated in the context of mixed reality interactive storytelling. The first part focuses on the description of the Artificial Intelligence (AI) mechanisms that are used to model and control the behavior of the application. We present an efficient real-time hierarchical planning engine, and show how active modalities (such as intentional gestures) and passive modalities (such as facial expressions) can be integrated into the planning algorithm, in such a way that the narrative (driven by the behavior of the virtual characters inside the virtual world) can effectively evolve in accordance with user interactions. The second part is devoted to the automatic recognition of user interactions. After briefly describing the implementation of a simple but robust rule-based gesture recognition system, the emphasis is set on facial expression recognition. A complete solution integrating state-of-the-art techniques along with original contributions is drawn. It includes face detection, facial feature extraction and analysis. The proposed approach combines statistical learning and probabilistic reasoning in order to deal with the uncertainty associated with the process of modeling facial expressions.

Recent progress in the overlay and registration of digital information on the users workspace in a spatially meaningful way has allowed mixed reality (MR) to become a more effective operational medium. In the area of medical surgery, surgeons are conveyed with information such as the incisions location, regions to be avoided, diseased tissues, etc, while staying in and keeping their original working environment. The main objective of this Thesis is identifying theoretical and practical basis for how mixed reality interfaces might provide support and augmentation maximizing the continuity of interaction. We start proposing a set of design principles organized in a design space which allows to identify continuity interaction properties at an early stage of the development system. Once the abstract design possibilities have been identified and a concrete design decision has been taken, an implementational strategy can be developed. Two approaches were investigated: markerless and marker-based. The last one is used to provide surgeons with guidance on an osteotomy task in the maxillo-facial surgery. The evaluation process applies usability tests with users to validate the augmented guidance in different scenarios and to study the influence of different design variables in the final user interaction. As a result we have found a model to describe the contribution factors of each variable for the continuity of the user interaction. We suggest that this methodology can be applied mainly to those applications in which smooth connections and interactions, with virtual and real environments, are critical for the system; i.e. surgery, drivers applications or pilot simulations.

Det här examensarbetet på D-nivå har utförts i samarbete men företaget XM reality i Linköping. Företaget jobbar med att ta fram olika system inom området mixed reality, som på svenska översätts till förhöjd verklighet.   Syftet med projektet var att utforma två olika handhållna projektorsystem. Produkterna skulle utformas för att passa in i sjukhusmiljö. Båda enheterna har samma funktion men innehåller olika komponenter. Produkterna är anpassade till en ny teknik som företaget har utvecklat. Den nya tekniken är en mixed reality applikation som gör det möjligt att projicera skiktscanningar, CT eller MRI, direkt på patientens kropp. Målet med projektet var att göra två fungerade modeller. Modellerna skall fungera som prototyper i den medicinska forskarstudien som projektet ingår i.   En designstrategi utformades till företaget som skall uttrycka företagets image och vision. I arbetet med designstrategin togs ett antal kärnvärden fram. Kärnvärdena blev high tech, professionell, framtid, exklusiv, kvalité, hållbar samt kompakt vilket skall gälla för företagets samtliga produkter. Till produktsegmentet sjukhusprodukter tillades kärnvärdena ergonomisk samt dynamisk.   Arbetet med att utveckla produkterna började med litterära ergonomiska studier som följdes upp av fysiska användarstudier. Under arbetats gång har flera olika metoder tillämpats. Metoderna har används för att fatta beslut men även som stöd i den kreativa gestaltningsprocessen.   Resultatet är två ergonomiskt anpassade handhållna enheter. Enheterna är utformade för att passa in i sjukhusmiljön. De är smidiga samt enkla att använda. Båda produkterna är utformade efter designstrategin och för att förmedla företagets image. De fysiska modeller som gjordes innehåller fungerande komponenter som går att bytas ut. / This thesis D-level is done in collaboration with the company XM reality that is located in Linköping, Sweden. The company is working with the development of various systems in the field of mixed reality.   The task was to design two different hand-held projector systems. The products should be designed to fit in to the hospital environment. Both products have the same function but contain different components. The products are adapted to a new technology that the company has developed. The new technology is a mixed reality application that makes it possible to project CT or MRI scans directly on to the patient's body. The goal of the project was to make two functioning models. The models are to be used as prototypes in the medical research study the project is a part of.   A design strategy was created for the company to reflect the company’s image and vision. In the process of developing the design strategy, a number of core values were established. The core values were high-tech, professional, futuristic, exclusive, quality, durable and compact, which should to be valid for all the company’s products. To the product segment of hospital products were the core values ergonomic and dynamic added.   The work to develop the products began with literary ergonomic studies that were followed up by user studies. During the working process, several different methods were used. The methods were used not only to make decisions but also to help the creative design process.   The result is two ergonomically designed hand-held devices. The products are designed to fit into the hospital environment. They are slim and easy to use. Both products are designed along the design strategy and to communicate the company's image. The physical models that were made contain functional components that can be changed.

D. Tech. Electrical Engineering. / Developes a wheelchair motion platform whereby its user may be introduced into a simulated world. This simulated world is then required to be closely related to real world spaces that will be encountered by a disabled person using a wheelchair as a mobility aid. The wheelchair to be accommodated in the simulation environment may have multiple mechanical construct possibilities. The wheelchair used on the simulation platform needs to be driven by a combination of two wheels, as is generally found on manual and electric wheelchairs. The final objective was to design the simulation as closely as possible to the real world in order to use the VS-1 motion platform for architectural evaluations, possible training and general research in the field of simulators used in an enabled environment.

This research examines mixed reality meetings, a context where individuals attend to both face-to-face group members while multitasking with technology. In these meetings, members engage simultaneously with those physically present and those outside of the meeting (virtual communication partners). Technology multitasking in meetings has a dual effect: it not only impacts the individual user, it has the potential to transform how collocated groups communicate and work together since attention becomes fragmented across multiple competing tasks. Qualitative and quantitative methods were used to investigate mixed reality meetings across four themes: (1) the factors contributing to the likelihood to multitask based on meeting type, polychronicity (one’s preference for multitasking), and cohesion beliefs, (2) behavior during mixed reality assessed by copresence management, (3) attitudes toward technology multitasking, and (4) subjective outcomes measured by perceived productivity and meeting satisfaction. The qualitative data set consists of fieldwork from a global software company and interviews with 8 information workers. The quantitative data are comprised of survey results from the fieldwork site (n=156) and an online panel of information workers (n=110). Results indicate that information workers perceive distinct meeting types that are associated with implicit norms for appropriate technology multitasking. These norms varied based on the relevance of a meeting segment and if a power figure was present. A higher preference score for multitasking (high polychronicity) was significantly correlated with increased technology multitasking and perceived productivity. Members of cohesive teams exhibited the most technology multitasking and perceived their teammates multitasking as appropriate. However, outsiders who exhibited the same behaviors were viewed as rude and distracting. Overall, information workers who multitasked during meetings did so with electronic communication tasks (e-mail and instant messaging) as opposed to other computing tasks (e.g. writing documents, researching information). These findings are discussed in relation to psychological studies on multitasking, computer-supported cooperative work, and social constructionist views of technology use. This dissertation is a contribution to the assessment of technology use in social settings, particularly in organizations where tasks are often interrupted and a reliance on electronic communication tools impacts how people manage and accomplish work. / text

Zugl.: Saarbrücken, Univ., Diss., 2006

Techn. Universiẗat, Diss., 2006--Darmstadt.

A manutenção industrial é um dos grandes desafios na área de automação e manufatura da indústria. Este desafio tem se apresentado, em âmbito nacional, pela adoção da prática “predizer e prevenir” ao invés de “falhar e consertar”. Enquanto o Brasil caminha para uma mudança de paradigma de manutenção, o cenário internacional se volta para soluções em manufatura sustentável que atendam aos requisitos “eco and green”. E a manutenção é parte crucial deste novo contexto de produção. Entretanto, somente nos anos noventa, a indústria européia começa a delinear estratégias de manutenção. A complexidade e multidisciplinaridade deste tema tem sido o principal obstáculo para melhoras substanciais na disciplina de manutenção. São nas estratégias de manutenção, aliadas ao crescente avanço em Tecnologias da Informação e Comunicação (TIC), que aparecem as possibilidades de superar as limitações e os desafios desta área de pesquisa. A Manutenção Inteligente (MI) é uma dessas estratégias e utiliza o paradigma baseado em condição (Condition-based maintenance) para aplicação da técnica “predizer e prevenir”. Contudo, uma das principais barreiras na adoção de MI é a sua transferência efetiva para o operador em chão de fábrica. O conhecimento gerado pelos sistemas MI deve ser transformado em informações inteligíveis e úteis para o operador. Entretanto, o acesso, o entendimento e o uso destes dados, durante a atividade de manutenção, não é um processo trivial, devido à sobrecarga de informações e sistemas envolvidos nestes processos. Surge, então, como alternativa para contornar este problema, o uso de técnicas de realidade mista como meio de potencializar a interface homem-máquina. A escolha, neste trabalho, por interfaces mistas, objetiva, não apenas a visualização, mas a integração e o gerenciamento da informação. A fim de alcançar estes objetivos, é desenvolvida uma abordagem baseada em modelo. A primeira etapa consiste na definição de um modelo conceitual que integre os diferentes domínios envolvidos na solução proposta. Nesta fase, foram identificadas três áreas da manutenção em que a realidade mista pode permitir um ganho significativo: diagnóstico, planejamento e segurança. A segunda etapa consiste na descrição formal de um modelo de dados para integrar e gerenciar informações de diversos formatos. Finalmente, é implementado um sistema de visualização para validar o modelo de dados em um conjunto de estudos de caso da indústria. Objetiva-se, com esta abordagem: (i) facilitar a identificação de quais componentes apresentam problemas, o que auxilia no processo de tomada de decisão nas tarefas de manutenção; (ii) a inclusão do human in the loop, com o sistema MI, em que o operador poderá solicitar dados e medições adicionais durante a atividade de manutenção; (iii) fornecer um modelo que integre dados de formatos distintos, que viabilize uma maneira fácil e flexível de configurar o ambiente virtual. A visualização mista, utilizada pelo operador de manutenção em chão de fábrica, pode representar uma vantagem competitiva para indústria, além de prover uma maior flexibilidade e segurança para o operador durante a atividade de manutenção. / Industrial maintenance is one of the major challenges in industrial automation and manufacturing area. This challenge presents itself on the national context by the transformation to "predict and prevent" rather than "fail and fix ". While the Brazil moves toward a paradigm shift of maintenance, the international context searches manufacturing sustainable solutions that reply according to eco and green requirements and the maintenance is crucial part of this new production environment. However, it was only in the nineties that European industry started discusses maintenance strategies. The maintenance complexity and the multidisciplinary have been the main limitations for substantial improvements in the maintenance discipline. There are maintenance strategies coupled with the advances in Information and Communication Technologies (ICT) that arise the possibilities to overcome the limitations and challenges of this research area. The intelligent maintenance (IM) is one of these strategies and it uses the Condition-Based Maintenance paradigm (CBM). On the other hand one of the obstacles about the IM adoption is its effective transfer to the operator on the factory floor. The knowledge from IM systems must be transformed in understandable information and useful to the operator. However, the access, the understanding and the use of such information during the maintenance activity are not a trivial activity due the data and systems overload involved in these processes. Thus, arises as an alternative to solve this problem, the using mixed reality techniques to human-computer interface. The use of mixed interfaces in this thesis will aim not only to visualization but the integration and management of information. To achieve these goals was developed a model-based approach. The first step consisted in defining a conceptual model that address the domains involved in the solution proposed. At this stage were identified three main areas where the mixed reality could allow a significant change to maintenance activities: diagnosis, planning and security. The second step consisted in the formal description of a data model for integrating and management of information from multiple formats. Finally a visualization system was implemented to validate the data model in a set of case studies of industry. The goals of this thesis are: (i) to facilitate the identification of which components present problems in order to help in the decision making process of the maintenance tasks; (ii) the inclusion of the "human in the loop" into intelligent maintenance system, where the operator can request measurements and additional information during the maintenance process; (iii) to supply a model to integrate data of different formats making available an easy and flexible tool for configuring the virtual environment. The mixed reality, used by the maintenance operator on factory floor, may represent a competitive advantage for industry and provide greater flexibility and safety for operator during maintenance activity.

abstract: Stroke is a leading cause of disability with varying effects across stroke survivors necessitating comprehensive approaches to rehabilitation. Interactive neurorehabilitation (INR) systems represent promising technological solutions that can provide an array of sensing, feedback and analysis tools which hold the potential to maximize clinical therapy as well as extend therapy to the home. Currently, there are a variety of approaches to INR design, which coupled with minimal large-scale clinical data, has led to a lack of cohesion in INR design. INR design presents an inherently complex space as these systems have multiple users including stroke survivors, therapists and designers, each with their own user experience needs. This dissertation proposes that comprehensive INR design, which can address this complex user space, requires and benefits from the application of interdisciplinary research that spans motor learning and interactive learning. A methodology for integrated and iterative design approaches to INR task experience, assessment, hardware, software and interactive training protocol design is proposed within the comprehensive example of design and implementation of a mixed reality rehabilitation system for minimally supervised environments. This system was tested with eight stroke survivors who showed promising results in both functional and movement quality improvement. The results of testing the system with stroke survivors as well as observing user experiences will be presented along with suggested improvements to the proposed design methodology. This integrative design methodology is proposed to have benefit for not only comprehensive INR design but also complex interactive system design in general. / Dissertation/Thesis / Doctoral Dissertation Media Arts and Sciences 2014

abstract: Emerging information and communication technology (ICT) has had an enormous effect on the building architecture, engineering, construction and operation (AECO) fields in recent decades. The effects have resonated in several disciplines, such as project information flow, design representation and communication, and Building Information Modeling (BIM) approaches. However, these effects can potentially impact communication and coordination of the virtual design contents in both design and construction phases. Therefore, and with the great potential for emerging technologies in construction projects, it is essential to understand how these technologies influence virtual design information within the organizations as well as individuals’ behaviors. This research focusses on understanding current emerging technologies and its impacts on projects virtual design information and communication among projects stakeholders within the AECO organizations. / Dissertation/Thesis / Doctoral Dissertation Civil and Environmental Engineering 2017