The overarching purpose of this research is to understand whether Mixed Reality can enhance a surgeon’s manipulations skills during minimally invasive procedures. Minimally-invasive surgery (MIS) utilizes small cuts in the skin - or sometimes natural orifices - to deploy instruments inside a patient’s body, while a live video feed of the surgical site is provided by an endoscopic camera and displayed on a screen. MIS is associated with many benefits: small scars, less pain and shorter hospitalization time as compared to traditional open surgery. However, these benefits come at a cost: because surgeons have to work by looking at a monitor, and not down on their own hands, MIS disrupts their eye-hand coordination and makes even simple surgical maneuvers challenging to perform. In this study, we wish to use Mixed Reality technology to superimpose anatomical models over the surgical site and explore if it can be used to mitigate this problem.

abstract: Mixed reality mobile platforms co-locate virtual objects with physical spaces, creating immersive user experiences. To create visual harmony between virtual and physical spaces, the virtual scene must be accurately illuminated with realistic physical lighting. To this end, a system was designed that Generates Light Estimation Across Mixed-reality (GLEAM) devices to continually sense realistic lighting of a physical scene in all directions. GLEAM optionally operate across multiple mobile mixed-reality devices to leverage collaborative multi-viewpoint sensing for improved estimation. The system implements policies that prioritize resolution, coverage, or update interval of the illumination estimation depending on the situational needs of the virtual scene and physical environment. To evaluate the runtime performance and perceptual efficacy of the system, GLEAM was implemented on the Unity 3D Game Engine. The implementation was deployed on Android and iOS devices. On these implementations, GLEAM can prioritize dynamic estimation with update intervals as low as 15 ms or prioritize high spatial quality with update intervals of 200 ms. User studies across 99 participants and 26 scene comparisons reported a preference towards GLEAM over other lighting techniques in 66.67% of the presented augmented scenes and indifference in 12.57% of the scenes. A controlled lighting user study on 18 participants revealed a general preference for policies that strike a balance between resolution and update rate. / Dissertation/Thesis / Masters Thesis Computer Science 2018

The temporal bone is an anatomically complex region within the skull. Current training for temporal bone surgery includes cadaveric, physical and virtual haptic simulations and apprenticeships. Cadavers are limited by low supply. Haptic devices are limited by their force and stiffness ratings and thus cannot adequately simulate rigid materials. Physical simulations excel at simulating stiff materials but do a poor job of soft tissue. The research objective was to develop a mixed reality (MR) temporal bone surgical haptic simulation. This novel concept would utilize physical models to simulate bone and haptic forces to simulate soft tissue. A surgical drill was attached to a Quanser® High Definition Haptic DeviceTM (HD2) via a clamp. An algorithm was implemented to simulate a force at the drill tip and to negate the weight of the clamp. This modified haptic system was interfaced to a temporal bone haptic simulation. Haptic chatter unique to the modified haptic system was observed and low-pass filters were used to mitigate this issue. Due to the poor positional accuracy of the HD2, MR simulation was not achieved. However, VR haptic simulation was achieved. Six expert surgeons were recruited to investigate the following questions: "What is the impact of different haptic hardware on surgical realism?" and "Would end users prefer a surgical drill over a standard haptic manipulandum?" Three cases were compared: a Phantom Omni®, a standard HD2 and a modified HD2 with attached drill. Expert surgeons rated the standard HD2 and Phantom Omni equivalently whilst preferring the modified HD2 with attached drill. Though the modified HD2 scored higher in all categories only “Acoustics” and “Overall Appreciation” displayed statistical significance. This implies that drill acoustics is critical for realism. / February 2016

No / The implementation of a mixed reality game using the J2ME platform is presented. The different levels of the game environment are presented and also the game navigation system which uses QR code and Bluetooth. Its relationship to other environments such as iPhone and Android, the portability issues, and the hardware and software specification for the game deployment are discussed. The game aims to stimulate and maintain a conversation between learners and their teachers, and also between learners and learners, whilst being immersed in the story of the game. It is found that the learners become more competent through enculturation, externalising their ideas, and socialising within their zone of proximal development.

Mixed Reality experiences, that blend physical and virtual objects, have become commonplace on handheld computing devices. One common application of these technologies is their use in cultural heritage "walking tours." These tours provide information about the surrounding environment in a variety of contexts, to suit the needs and interests of different groups of participants. Using the familiar "campus tour" as a canonical example, this dissertation investigates the technical and cognitive processes involved in transferring this tour from its physical and analog form into Mixed Reality. Using the concept of spatial scale borrowed from cognitive geography, this work identifies the need to create and maintain continuity across different scales of spatial experience as being of paramount importance to successful Mixed Reality walking tours. The concepts of scale transitions, coordination of representations across scales, and scale-matching are shown to be essential to maintaining the continuity of experience. Specific techniques that embody these concepts are also discussed and demonstrated in a number of Mixed Reality examples, including in the context of a successful deployment of a Mixed Reality Tour of the Georgia Tech campus. The potential for a "Language of Mixed Reality" based on the concepts outlined in this work is also discussed, and a general framework, called the Mixed Reality Scale Framework is shown to meet all the necessary criteria for being a cognitive theory of Human-Centered Computing in the context of Mixed Reality.

Background. With the growing prospects of extended realities (XR), new use casesand experiences are constantly being developed. Especially with the introduction ofmixed reality (MR), allowing for a more seamless blend of the physical and digitalspace, it provides great opportunities in many fields such as education and trainingwhere dangerous procedures can be practiced safely. However, to make these experi-ences as effective and educational as possible, there is a need to make the experiencesrealistic. Objectives. One important aspect of creating realistic experiences is believablecollision between the user’s physical hand and the digital objects. This study specif-ically takes aim at this aspect. Trying to find how the performance difference anduser experience (UX) is affected by the addition of collision around the user’s handsin an MR environment. In order to help guide the way to get the answers to thesequestions, a set of objectives has been formulated. These objectives are; finding andimplementing a hand collision method, designing and performing the user study, andfinally finding and utilizing appropriate methods for analyzing the collected data. Methods. To get a better understanding of the UX and performance of using handcollision, a user study was created where the participants had to complete a seriesof tasks, with and without collision around their hands. For each task, answering aquestionnaire about their experience. Once the data have been collected, it will beanalyzed with the help of the SUS scoring system and statistical tests. Results. The study had 12 participants. With and without hand collision receivedan average SUS score of 62,5 and 69,2 respectively. The results show that the methodusing no collision performed better in terms of time to complete the task. However,hand collision performed better with fewer grabs used. No statistically significantdifference was detected between having or not having hand collision in terms of in-tuitiveness and realism. However, participants were observed to intuitively use thehand collision to their advantage. Conclusions. In conclusion, the participants did not perform better with handcollision, however, did indicate some level of increased intuition and realism. Thenegative aspects of the hand collision are believed to be attributed to the methodused to implement it, and potential in the area exists for further improvements andresearch. / Bakgrund. Med ett växande potential för extended realities (XR), nya använd-ningsområden och upplevelser utvecklas ständigt. Speciellt med införandet av mixedrealities (MR), möjligjorde en mer enad upplevelse av det fysiska och digitala, medstora möjligheter inom utbildning och träning där det farligt sitvationer kan övas påett säkert sätt. Men, för att göra dessa upplevelser så effektiva och pedagogiska sommöjligt behöves mer realistiska upplevelser. Syfte. En viktig aspekt av att skapa realistiska upplevelser är att skapa trovärdigakollisioner mellan användarens fysiska hand och the digitala objekten. Detta är ettav målen denna studien tar sikte på. Att försöker hitta hur prestandaskillnaden äroch användarupplevelsen (UX) påverkas med tillägget av kollision runt användarenshänder i en MR-miljö. För att enklare kunna hitta en väg till svaret för dessa frå-gor har mål formulerats. Dessa mål är att; hitta och implementera en handkollisionsmetod, designa och utför en användarstudie, och hitta samt använd lämpliga metoderför att analysera den insamlade datan. Metod. För att få en bättre förståelse för hur UX och prestanda för använd-ning av handkollsion skapades en användarstudie där deltagarna genomförde en serieuppgifter, med och utan kollision runt deras händer. För varje uppgift besvaradesett frågeformulär om deras upplevelse. När uppgifterna har samlats in kommer deatt analyseras med hjälp av SUS poängsystem och statistiska tester. Resultat. Denna studie hade 12 deltagare. Med och utan handkollision fick engenomsnittlig SUS-poäng av 62,5 respektive 69,2. Resultaten visar att metoden sominte använder någon kollision presterade bättre när det gäller tid för att slutförauppgiften. Men, kollision fick dock bättre med resultat med ett färre antal greppsom används. Det var ingen statistiskt signifikant skillnad som upptäcktes mellanmed och utan handkollision i avsikt på intuitivitet och realism. Dock observeradesdeltagarna att använda kollisionen på ett mer intuitivt sätt till sin fördel. Slutsatser. Sammanfattningsvis, deltagarna presterade inte bättre med kollision,men indikerade viss nivå av ökad intuition och realism. De negativa aspekterna avkollisionen tros att vara i grund på den metod som använts för att implementera den,och potential finns inom området för ytterligare förbättringar och forskning.

The need to be economically successful is the key driver for companies to be innovative and implement new technologies with increasing efficiency and effectiveness. Uncertainty about whether to use new technologies and missing knowledge about their advantages lead to staggering and withholding from fast diffusion of innovations. Focusing on the industry of logistics and the technology Mixed Reality, this research project developed a methodical approach for evaluating the fitness of an innovative technology and a specific process of application. A mixed methods approach was derived, based on interviews and experiments. The main methodologies used, were semi-structured interviews with decision makers in logistics companies to elaborate triggering criteria in the investment process and laboratory experiments for the evaluation of competing technologies. These methods were framed by an initial field experiment and feedback interviews after the analysis for the validation of the approach. The research proved the competitively viable applicability of Mixed Reality and its specific strengths and weaknesses in manual order picking. This set the foundation for possible further development and implementation of the technology. The developed methodological approach proved to be a valid and reliable assessment of the intersection between a technology and specified process of application. This can greatly enhance the speed of implementing new innovations and gaining competitive advantages for companies.

This thesis presents a real-time automated building progress monitoring solution for indoor environments using a mobile device. Such a system could prompt accurate and timely assessment of work progress that would allow managers to make adjustments and minimise both time and cost overrun when deviations from the schedule occur. Although many researchers have proposed approaches for progress monitoring in outdoor scenes, these cannot perform in real-time and shift into the complex interior environment. Research efforts for indoor environments are not fully automated and lead to errors in more complex scenes. Systems based on mobile devices could potentially enhance the inspection process and reduce the required time by allowing the inspector to acquire progress data by simply walking around the site. The main challenge of these systems is the tracking of the pose of the camera to achieve accurate alignment between the 3D design model and the real-world scene. Methods for estimating the user’s pose rely on a) tags on each target of interest, which require additional time and cost for installation and maintenance; b) pre-selected user locations, which restricts the user to those locations only; or c) GPS on the augmented reality headset, which only applies to outdoor inspections. Additionally, current mobile-based inspection systems do not perform any comparison between the captured as-built and the as-planned data. In this research, different marker-less Augmented Reality (AR) potential methods were implemented and tested for finding the most robust tracking solution. The Microsoft HoloLens was found to be the top performer for tracking the user’s pose and for overall user-experience. Next, a semi-automated method was developed for initially registering the 3D model to the real environment by exploiting information from detected floor and wall surfaces. Results showed that this method reduces the time of the initial registration by 58%. Having the 3D model aligned to the real environment and knowing the pose of the camera at every moment, an automated method was developed that exploits the captured as-built surface mesh data from the mobile device, compares it against the 3D design model and identifies in real-time whether an object has been built according to plan. Different parameters were tested for finding the optimum combination based on the current quality of mesh data. If quality of mesh data changes, then new parameters should be explored. Finally, the proposed solution was tested in real site conditions resulting in 76.6% precision, 100.0% recall, and 83.5% accuracy.

Mixed Reality (MR) is a technique to blend together the real life with virtual reality. Using this technique it is, for instance, possible for experts to assist persons several miles away to perform tasks by talking and  visually aid them. In this thesis the main issue is to see how the delay in such a system for remote assistance eects the users. A controlled test was carried out with 20 test persons of dierent backgrounds. The study shows that it is very likely to be able to use an MR system for remote assistance even if there is a delay between the user and the expert. As long as they both are aware of the problem and are able to take it easy and do not have to move around too much it is still possible to work with delays to up to 4000 ms. Furthermore, the average time of completion for a task did not increase with the added delay. It was linear, i.e. the task is not more difficult toperform when the instructions are delayed.

This performance-based project scrutinizes indeterminacy as a mediating force impinging upon our behaviour and its subsequent impact on the nature and constituency of engagements and dialogue between people in urban spaces. Concepts centering on the dynamics of departure are being investigated with focus upon the Multi User Virtual Environment (MUVE), Second Life as a facet of real life (the term 'real life' will henceforth be referred to in this document within the context of Second Life, as First Life). Experienced through the vehicle of the Roaming Body, our meetings and encounters with people and places frequently manifest as disjunct communiqués and mis-engagements. I am asserting that this is due to the inevitability in our existence of indeterminacy acting as a significant factor in the articulation of our relations with others, reinforcing our description as time-based entities traversing the passage of the everyday. I maintain that this is frequently evidenced in our behaviours through the occurrence (notwithstanding arrivals) of a continual, pre-emptive state of departure. Indeterminacy implies motion and emerges, as Massumi asserts, through ‘… an unfolding relation to its own nonpresent potential to vary …’. We, as humans, are constantly being drawn away – always either approaching or embracing involuntarily, a 'state-of-Leaving’ which co-mingles with and unerringly erodes our efforts to stay engaged with another in the here and now. In my dance and video practice, interventionist dance strategies are being used to prompt and interrogate the constituents of departure within encounters in designated public places. Experimental movement frameworks employed are informed by the discipline of Contact Improvisation Dance and Authentic Movement. The working process is being documented using a range of video narrative.