abstract: Virtual Reality (hereafter VR) and Mixed Reality (hereafter MR) have opened a new line of applications and possibilities. Amidst a vast network of potential applications, little research has been done to provide real time collaboration capability between users of VR and MR. The idea of this thesis study is to develop and test a real time collaboration system between VR and MR. The system works similar to a Google document where two or more users can see what others are doing i.e. writing, modifying, viewing, etc. Similarly, the system developed during this study will enable users in VR and MR to collaborate in real time. The study of developing a real-time cross-platform collaboration system between VR and MR takes into consideration a scenario in which multiple device users are connected to a multiplayer network where they are guided to perform various tasks concurrently. Usability testing was conducted to evaluate participant perceptions of the system. Users were required to assemble a chair in alternating turns; thereafter users were required to fill a survey and give an audio interview. Results collected from the participants showed positive feedback towards using VR and MR for collaboration. However, there are several limitations with the current generation of devices that hinder mass adoption. Devices with better performance factors will lead to wider adoption. / Dissertation/Thesis / Final Output / Keynote / Thesis Demo / Masters Thesis Computer Science 2017

A manutenção industrial é um dos grandes desafios na área de automação e manufatura da indústria. Este desafio tem se apresentado, em âmbito nacional, pela adoção da prática “predizer e prevenir” ao invés de “falhar e consertar”. Enquanto o Brasil caminha para uma mudança de paradigma de manutenção, o cenário internacional se volta para soluções em manufatura sustentável que atendam aos requisitos “eco and green”. E a manutenção é parte crucial deste novo contexto de produção. Entretanto, somente nos anos noventa, a indústria européia começa a delinear estratégias de manutenção. A complexidade e multidisciplinaridade deste tema tem sido o principal obstáculo para melhoras substanciais na disciplina de manutenção. São nas estratégias de manutenção, aliadas ao crescente avanço em Tecnologias da Informação e Comunicação (TIC), que aparecem as possibilidades de superar as limitações e os desafios desta área de pesquisa. A Manutenção Inteligente (MI) é uma dessas estratégias e utiliza o paradigma baseado em condição (Condition-based maintenance) para aplicação da técnica “predizer e prevenir”. Contudo, uma das principais barreiras na adoção de MI é a sua transferência efetiva para o operador em chão de fábrica. O conhecimento gerado pelos sistemas MI deve ser transformado em informações inteligíveis e úteis para o operador. Entretanto, o acesso, o entendimento e o uso destes dados, durante a atividade de manutenção, não é um processo trivial, devido à sobrecarga de informações e sistemas envolvidos nestes processos. Surge, então, como alternativa para contornar este problema, o uso de técnicas de realidade mista como meio de potencializar a interface homem-máquina. A escolha, neste trabalho, por interfaces mistas, objetiva, não apenas a visualização, mas a integração e o gerenciamento da informação. A fim de alcançar estes objetivos, é desenvolvida uma abordagem baseada em modelo. A primeira etapa consiste na definição de um modelo conceitual que integre os diferentes domínios envolvidos na solução proposta. Nesta fase, foram identificadas três áreas da manutenção em que a realidade mista pode permitir um ganho significativo: diagnóstico, planejamento e segurança. A segunda etapa consiste na descrição formal de um modelo de dados para integrar e gerenciar informações de diversos formatos. Finalmente, é implementado um sistema de visualização para validar o modelo de dados em um conjunto de estudos de caso da indústria. Objetiva-se, com esta abordagem: (i) facilitar a identificação de quais componentes apresentam problemas, o que auxilia no processo de tomada de decisão nas tarefas de manutenção; (ii) a inclusão do human in the loop, com o sistema MI, em que o operador poderá solicitar dados e medições adicionais durante a atividade de manutenção; (iii) fornecer um modelo que integre dados de formatos distintos, que viabilize uma maneira fácil e flexível de configurar o ambiente virtual. A visualização mista, utilizada pelo operador de manutenção em chão de fábrica, pode representar uma vantagem competitiva para indústria, além de prover uma maior flexibilidade e segurança para o operador durante a atividade de manutenção. / Industrial maintenance is one of the major challenges in industrial automation and manufacturing area. This challenge presents itself on the national context by the transformation to "predict and prevent" rather than "fail and fix ". While the Brazil moves toward a paradigm shift of maintenance, the international context searches manufacturing sustainable solutions that reply according to eco and green requirements and the maintenance is crucial part of this new production environment. However, it was only in the nineties that European industry started discusses maintenance strategies. The maintenance complexity and the multidisciplinary have been the main limitations for substantial improvements in the maintenance discipline. There are maintenance strategies coupled with the advances in Information and Communication Technologies (ICT) that arise the possibilities to overcome the limitations and challenges of this research area. The intelligent maintenance (IM) is one of these strategies and it uses the Condition-Based Maintenance paradigm (CBM). On the other hand one of the obstacles about the IM adoption is its effective transfer to the operator on the factory floor. The knowledge from IM systems must be transformed in understandable information and useful to the operator. However, the access, the understanding and the use of such information during the maintenance activity are not a trivial activity due the data and systems overload involved in these processes. Thus, arises as an alternative to solve this problem, the using mixed reality techniques to human-computer interface. The use of mixed interfaces in this thesis will aim not only to visualization but the integration and management of information. To achieve these goals was developed a model-based approach. The first step consisted in defining a conceptual model that address the domains involved in the solution proposed. At this stage were identified three main areas where the mixed reality could allow a significant change to maintenance activities: diagnosis, planning and security. The second step consisted in the formal description of a data model for integrating and management of information from multiple formats. Finally a visualization system was implemented to validate the data model in a set of case studies of industry. The goals of this thesis are: (i) to facilitate the identification of which components present problems in order to help in the decision making process of the maintenance tasks; (ii) the inclusion of the "human in the loop" into intelligent maintenance system, where the operator can request measurements and additional information during the maintenance process; (iii) to supply a model to integrate data of different formats making available an easy and flexible tool for configuring the virtual environment. The mixed reality, used by the maintenance operator on factory floor, may represent a competitive advantage for industry and provide greater flexibility and safety for operator during maintenance activity.

abstract: Millions of Americans live with motor impairments resulting from a stroke and the best way to administer rehabilitative therapy to achieve recovery is not well understood. Adaptive mixed reality rehabilitation (AMRR) is a novel integration of motion capture technology and high-level media computing that provides precise kinematic measurements and engaging multimodal feedback for self-assessment during a therapeutic task. The AMRR system was evaluated in a small (N=3) cohort of stroke survivors to determine best practices for administering adaptive, media-based therapy. A proof of concept study followed, examining changes in clinical scale and kinematic performances among a group of stroke survivors who received either a month of AMRR therapy (N = 11) or matched dosing of traditional repetitive task therapy (N = 10). Both groups demonstrated statistically significant improvements in Wolf Motor Function Test and upper-extremity Fugl-Meyer Assessment scores, indicating increased function after the therapy. However, only participants who received AMRR therapy showed a consistent improvement in their kinematic measurements, including those measured in the trained reaching task (reaching to grasp a cone) and in an untrained reaching task (reaching to push a lighted button). These results suggest that that the AMRR system can be used as a therapy tool to enhance both functionality and reaching kinematics that quantify movement quality. Additionally, the AMRR concepts are currently being transitioned to a home-based training application. An inexpensive, easy-to-use, toolkit of tangible objects has been developed to sense, assess and provide feedback on hand function during different functional activities. These objects have been shown to accurately and consistently track hand function in people with unimpaired movements and will be tested with stroke survivors in the future. / Dissertation/Thesis / Ph.D. Bioengineering 2012

A manutenção industrial é um dos grandes desafios na área de automação e manufatura da indústria. Este desafio tem se apresentado, em âmbito nacional, pela adoção da prática “predizer e prevenir” ao invés de “falhar e consertar”. Enquanto o Brasil caminha para uma mudança de paradigma de manutenção, o cenário internacional se volta para soluções em manufatura sustentável que atendam aos requisitos “eco and green”. E a manutenção é parte crucial deste novo contexto de produção. Entretanto, somente nos anos noventa, a indústria européia começa a delinear estratégias de manutenção. A complexidade e multidisciplinaridade deste tema tem sido o principal obstáculo para melhoras substanciais na disciplina de manutenção. São nas estratégias de manutenção, aliadas ao crescente avanço em Tecnologias da Informação e Comunicação (TIC), que aparecem as possibilidades de superar as limitações e os desafios desta área de pesquisa. A Manutenção Inteligente (MI) é uma dessas estratégias e utiliza o paradigma baseado em condição (Condition-based maintenance) para aplicação da técnica “predizer e prevenir”. Contudo, uma das principais barreiras na adoção de MI é a sua transferência efetiva para o operador em chão de fábrica. O conhecimento gerado pelos sistemas MI deve ser transformado em informações inteligíveis e úteis para o operador. Entretanto, o acesso, o entendimento e o uso destes dados, durante a atividade de manutenção, não é um processo trivial, devido à sobrecarga de informações e sistemas envolvidos nestes processos. Surge, então, como alternativa para contornar este problema, o uso de técnicas de realidade mista como meio de potencializar a interface homem-máquina. A escolha, neste trabalho, por interfaces mistas, objetiva, não apenas a visualização, mas a integração e o gerenciamento da informação. A fim de alcançar estes objetivos, é desenvolvida uma abordagem baseada em modelo. A primeira etapa consiste na definição de um modelo conceitual que integre os diferentes domínios envolvidos na solução proposta. Nesta fase, foram identificadas três áreas da manutenção em que a realidade mista pode permitir um ganho significativo: diagnóstico, planejamento e segurança. A segunda etapa consiste na descrição formal de um modelo de dados para integrar e gerenciar informações de diversos formatos. Finalmente, é implementado um sistema de visualização para validar o modelo de dados em um conjunto de estudos de caso da indústria. Objetiva-se, com esta abordagem: (i) facilitar a identificação de quais componentes apresentam problemas, o que auxilia no processo de tomada de decisão nas tarefas de manutenção; (ii) a inclusão do human in the loop, com o sistema MI, em que o operador poderá solicitar dados e medições adicionais durante a atividade de manutenção; (iii) fornecer um modelo que integre dados de formatos distintos, que viabilize uma maneira fácil e flexível de configurar o ambiente virtual. A visualização mista, utilizada pelo operador de manutenção em chão de fábrica, pode representar uma vantagem competitiva para indústria, além de prover uma maior flexibilidade e segurança para o operador durante a atividade de manutenção. / Industrial maintenance is one of the major challenges in industrial automation and manufacturing area. This challenge presents itself on the national context by the transformation to "predict and prevent" rather than "fail and fix ". While the Brazil moves toward a paradigm shift of maintenance, the international context searches manufacturing sustainable solutions that reply according to eco and green requirements and the maintenance is crucial part of this new production environment. However, it was only in the nineties that European industry started discusses maintenance strategies. The maintenance complexity and the multidisciplinary have been the main limitations for substantial improvements in the maintenance discipline. There are maintenance strategies coupled with the advances in Information and Communication Technologies (ICT) that arise the possibilities to overcome the limitations and challenges of this research area. The intelligent maintenance (IM) is one of these strategies and it uses the Condition-Based Maintenance paradigm (CBM). On the other hand one of the obstacles about the IM adoption is its effective transfer to the operator on the factory floor. The knowledge from IM systems must be transformed in understandable information and useful to the operator. However, the access, the understanding and the use of such information during the maintenance activity are not a trivial activity due the data and systems overload involved in these processes. Thus, arises as an alternative to solve this problem, the using mixed reality techniques to human-computer interface. The use of mixed interfaces in this thesis will aim not only to visualization but the integration and management of information. To achieve these goals was developed a model-based approach. The first step consisted in defining a conceptual model that address the domains involved in the solution proposed. At this stage were identified three main areas where the mixed reality could allow a significant change to maintenance activities: diagnosis, planning and security. The second step consisted in the formal description of a data model for integrating and management of information from multiple formats. Finally a visualization system was implemented to validate the data model in a set of case studies of industry. The goals of this thesis are: (i) to facilitate the identification of which components present problems in order to help in the decision making process of the maintenance tasks; (ii) the inclusion of the "human in the loop" into intelligent maintenance system, where the operator can request measurements and additional information during the maintenance process; (iii) to supply a model to integrate data of different formats making available an easy and flexible tool for configuring the virtual environment. The mixed reality, used by the maintenance operator on factory floor, may represent a competitive advantage for industry and provide greater flexibility and safety for operator during maintenance activity.

Human anatomy is traditionally taught using textbooks and dissections. With the advent of computer graphics, using 3D applications have started to see much more use in medical educations around the world. Today, technology such as Augmented Reality and Virtual Reality are on everybody’s lips, and many are now curious what we can do with this new technology. This thesis explores how Augmented Reality can be used in medical education to teach human anatomy. Two application prototypes were developed for the Microsoft Hololens that together tests different ways to use and interact with Augmented Reality. These applications were then tested in a case study with six medical students. From this study a number of hypotheses were formulated.

Cette thèse s'intéresse à la plasticité des interfaces de Réalité Mixte (RM), c'est-à-dire les applications de Réalité Virtuelle (RV), Réalité Augmentée (RA) et de Virtualité Augmentée (AV). Il y a un réel engouement aujourd’hui pour ce type d’applications notamment grâce à la démocratisation des périphériques tels les lunettes et casques immersifs, les caméras de profondeur et les capteurs de mouvement. La Réalité Mixte trouve notamment ses usages dans le divertissement, la visualisation de données, la formation et la conception en ingénierie. La plasticité d'un système interactif est sa capacité à s'adapter aux contraintes matérielles et environnementales dans le respect de son utilisabilité. La continuité de l'utilisabilité d'une interface plastique est assurée quel que soit le contexte d'usage. Nous proposons ainsi des modèles et une solution logicielle nommée 3DPlasticToolkit afin de permettre aux développeurs de créer des interfaces de Réalité Mixtes plastiques. Tout d'abord, nous proposons trois modèles pour modéliser les sources d'adaptation : un modèle pour représenter les dispositifs d'interaction et les dispositifs d'affichage, un modèle pour représenter les utilisateurs et leurs préférences et un modèle pour représenter la structure et la sémantique des données. Ces sources d'adaptation vont être prises en compte par un processus d'adaptation qui va déployer dans une application les composants applicatifs adaptés au contexte d'usage grâce à des mécanismes de notation. Le déploiement de ces composants va permettre d'adapter à la fois les techniques d'interaction de l'application et également la présentation de son contenu. Nous proposons également un processus de redistribution qui va permettre à l'utilisateur final de changer la distribution des composants de son système sur différentes dimensions : affichage, utilisateur et plateforme. Ce processus va ainsi permettre à l'utilisateur de changer de plateforme dynamiquement ou encore de combiner plusieurs plateformes. L'implémentation de ces modèles dans 3DPlasticToolkit permet de fournir aux développeurs une solution prête à l'usage qui peut gérer les périphériques actuels de Réalité Mixte et qui inclut un certain nombre de techniques d'interaction, d'effets visuels et de métaphores de visualisation de données. / This PhD thesis focuses on plasticity for Mixed Reality (MR) User interfaces, which includes Virtual Reality (VR), Augmented Reality (AR) and Augmented Virtuality (AV) applications. Today, there is a growing interest for this kind of applications thanks to the generalization of devices such as Head Mounted Displays, Depth sensors and tracking systems. Mixed Reality application can be used in a wide variety of domains such as entertainment, data visualization, education and training, and engineering. Plasticity refers to the capacity of an interactive system to withstand variations of both the system physical characteristics and the environment while preserving its usability. Usability continuity of a plastic interface is ensured whatever the context of use. Therefore, we propose a set of software models, integrated in a software solution named 3DPlasticToolkit, which allow any developer to create plastic MR user interfaces. First, we propose three models for modeling adaptation sources: a model for the description of display devices and interaction devices, a model for the description of the users and their preferences, a model for the description of data structure and semantic. These adaptation sources are taken into account by an adaptation process that deploys application components adapted to the context of use thanks to a scoring system. The deployment of these application components lets the system adapt the interaction techniques of the application of its content presentation. We also propose a redistribution process that allows the end-user to change the distribution of his/her application components across multiple dimensions: display, user and platform. Thus, it allows the end-user to switch dynamically of platform or to combine multiple platforms. The implementation of these models in 3DPlasticToolkit provides developers with a ready to use solution for the development of plastic MR user interfaces. Indeed, the solution already integrates different display devices and interaction devices and also includes multiple interaction techniques, visual effects and data visualization metaphors.

Tangible interfaces and mixed-reality environments have potential to bring together the advantages of physical and virtual environments to improve children’s learning and enjoyment. However, there are too few controlled experiments that investigate whether interacting with physical objects in the real world accompanied by interactive feedback may actually improve student learning compared to flat-screen interaction. Furthermore, we do not have a sufficient empirical basis for understanding how a mixed-reality environment should be designed to maximize learning and enjoyment for children. I created EarthShake, a mixed-reality game bridging physical and virtual worlds via a Kinect depth-camera and a specialized computer vision algorithm to help children learn physics. I have conducted three controlled experiments with EarthShake that have identified features that are more and less important to student learning and enjoyment. The first experiment examined the effect of observing physical phenomena and collaboration (pairs versus solo), while the second experiment replicated the effect of observing physical phenomena while also testing whether adding simple physical control, such as shaking a tablet, improves learning and enjoyment. The experiments revealed that observing physical phenomena in the context of a mixed-reality game leads to significantly more learning (5 times more) and enjoyment compared to equivalent screen-only versions, while adding simple physical control or changing group size (solo or pairs) do not have significant effects. Furthermore, gesture analysis provides insight as to why experiencing physical phenomena may enhance learning. My thesis work further investigates what features of a mixed-reality system yield better learning and enjoyment, especially in the context of limited experimental results from other mixed-reality learning research. Most mixed-reality environments, including tangible interfaces (where users manipulate physical objects to create an interactive output), currently emphasize open-ended exploration and problem solving, and are claimed to be most effective when used in a discovery-learning mode with minimal guidance. I investigated how critical to learning and enjoyment interactive guidance and feedback is (e.g. predict/observe/explain prompting structure with interactive feedback), in the context of EarthShake. In a third experiment, I compared the learning and enjoyment outcomes of children interacting with a version of EarthShake that supports guided-discovery, another version that supports exploration in discovery-learning mode, and a version that is a combination of both guideddiscovery and exploration. The results of the experiment reveals that Guided-discovery and Combined conditions where children are exposed to the guided discovery activities with the predict-observe-explain cycle with interactive feedback yield better explanation and reasoning. Thus, having guided-discovery in a mixed-reality environment helps with formulating explanation theories in children’s minds. However, the results also suggest that, children are able to activate explanatory theory in action better when the guided discovery activities are combined with exploratory activities in the mixed-reality system. Adding exploration to guided-discovery activities, not only fosters better learning of the balance/physics principles, but also better application of those principles in a hands-on, constructive problem-solving task. My dissertation contributes to the literatures on the effects of physical observation and mixed-reality interaction on students’ science learning outcomes in learning technologies. Specifically, I have shown that a mixed-reality system (i.e., combining physical and virtual environments) can lead to superior learning and enjoyment outcomes than screen-only alternatives, based on different measures. My work also contributes to the literature of exploration and guided-discovery learning, by demonstrating that having guided-discovery activities in a mixed-reality setting can improve children’s fundamental principle learning by helping them formulate explanations. It also shows that combining an engineering approach with scientific thinking practice (by combining exploration and guided-discovery activities) can lead to better engineering outcomes such as transferring to constructive hands-on activities in the real world. Lastly, my work aims to make a contribution from the design perspective by creating a new mixed-reality educational system that bridges physical and virtual environments to improve children’s learning and enjoyment in a collaborative way, fostering productive dialogue and scientific curiosity in museum and school settings, through an iterative design methodology to ensure effective learning and enjoyment outcomes in these settings.

This project aims to explore what the brand Land Rover could stand for in the future. The brands rich history of exploring unconquered terrain earned it admiration and desirability all around the world. Further extending it's reach onto new worlds is within reach. In the 2030s the first manned missions to Mars are planned. The first arrivers will have exploration vehicles, that are limited in range and capability. To really explore the planet, vehicles with greater off-road capability and range will be needed. The vehicles also need to allow the expedition crews to stay in the vehicle for longer periods comfortably and also offer extended life support on multi-week long journeys.With this project I am exploring possible answers to face the harsh conditions on Mars. Furthermore, the vehicle and it's features project a vision of what a future off-road driving experience could be.

A estimação da iluminação é essencial para aplicações de realidade mista que se propõem a integrar elementos virtuais a cenas reais de maneira harmoniosa e sem a perda do realismo. Um dos métodos mais utilizados para fazer essa estimação é conhecido como iluminação baseada em imagens (Image Based Lighting - IBL), método que utiliza light probes para capturar a intensidade da iluminação incidente em uma cena. Porém, IBL estima a iluminação incidente apenas para um determinado instante e posição. Nesta dissertação, será avaliado um modelo de iluminação que utiliza séries temporais de imagens de light probes, obtidas de maneira esparsa em relação ao tempo, para renderizar cenas em instantes arbitrários. Novas cenas contendo objetos virtuais poderão ser renderizadas utilizando imagens de light probes artificiais, geradas a partir das amostras da iluminação originais. Diferentes funções de interpolação e aproximação são avaliadas para modelar o comportamento luminoso. As imagens finais produzidas pela metodologia também são verificadas por voluntários, de modo a determinar o impacto na qualidade de renderização em aplicações de realidade mista. Além da metodologia, foi desenvolvida uma ferramenta de software em forma de plugin para facilitar o uso de IBL temporalmente variável, permitindo assim a renderização realística de objetos virtuais para instantes arbitrários / Lighting estimation is essential for mixed reality applications that strive to integrate virtual elements into real scenes in a seamlessly fashion without sacrificing realism. A widely used method for lighting estimation is known as Image Based Lighting (IBL), which utilizes light probes to determine incident light intensity within a scene. However, IBL estimates light incidence only for a given time and position. In this dissertation, we assess a lighting model based on a time series of light probe images, obtained sparsely, to render scenes at arbitrary times. New scenes containing virtual objects can then be rendered by using artificial light probe images, which are generated from the original light samples. Different types of interpolation and approximation functions were evaluated for modeling lighting behavior. The resulting images were assessed for the impact in rendering quality for mixed reality applications by volunteers. In addition to the methodology, we also developed a software plugin to simplify the use of temporally variable IBL, allowing realistic rendering of virtual objects for arbitrary times

Diminished reality, as a sub-topic of augmented reality where digital information is overlaid on an environment, is the perceived removal of an object from an environment. Previous approaches to diminished reality used digital replacement techniques, inpainting, and multi-view homographies. However, few used a virtual representation of the real environment, limiting their domains to planar environments. This thesis provides a framework to achieve real-time diminished reality on an augmented reality headset. Using state-of-the-art hardware, we combine a virtual representation of the real environment with inpainting to remove existing objects from complex environments. Our work is found to be competitive with previous results, with a similar qualitative outcome under the limitations of available technology. Additionally, by implementing new texturing algorithms, a more detailed representation of the real environment is achieved.