Nowadays, robots have become of high importance within the manufacturing industry, as well as programming methods have evolved to adapt to the current necessities. Due to the great number of complex concepts in Robotics, the learning curve of actual robotic software can be steep for inexperienced programmers, and previous training is required. However, training that requires the use of real robots is not always available for common users because of the difficult access to the machines. In parallel, new and more powerful Mixed Reality (MR) devices have been developed, expanding the possibilities of path planning. In this project, a MR application to support robot programming for welding purposes is developed, where the targets are placed according to a work object and a welding piece. Within this application, the user can interact with several menus and objects to program the path, which can be configured and simulated in the virtual environment with reachability checking. Paths can also be exported to the real robot, where they are conducted with accurate performance. Considering the learning process, a guide can be activated to show recommended steps, and it is possible to jog the virtual robot to comprehend robot motions. In addition, two experiments are conducted to test the spatial mapping and marker detection capabilities. Finally, the benefits, limitations, and flexibility of the application are discussed considering its impact on humans, while the user experience is evaluated with a survey for further analysis. The resulting application allows the user to generate, configure and export paths through the MR environment and test them in a real robot.

Two common problems in Remote Guidance applications include the remote guides lack of direct control over their view into the worker’s physical environment and the difficulties that arise with trying to place virtual 3D objects in a real 3D environment,via a moving, shaky, 2D image.The first issue can be called a lack of remote spatial awareness, the guide can see only what the worker enables them to see. In the worst case the guide is rendered blind to the task environment while the worker is unable to use their device. A common occurrence is tasks that require both hands.The second issue arises from the inherent difficulty present in trying to correctly place a 3D object using only a limited perspective. Camera shake and unreliable tracking of the physical environment being depicted only further add to this problem. Studies show that 3D annotations make for much more effective means of communication, especially in 3D task environments. Allowing the guide some measure of control over their own view has also been shown to improve the guides ability to aid their partner. This paper investigates a method of Remote Guidance where the task of environment tracking and object placement are separated. A prototype application is developed and tested against a baseline 2D-annotation Remote Guidance tool. The study finds the prototype to be an effective way of placing virtual 3D objects in a remote environment. Experimental results show that communication is indeed made better by the inclusion of 3D objects into Remote Guidance. This comes at the cost of a slight increase in the timetaken to complete a task as the complexity of the 3D tool is greater than the 2D one. Unfortunately, the experiment performed fails to properly account for remote spatial awareness.

There are different ways to interact with different hardware, therefore it is important to have an understanding about what factors that affect the experience when designing interactions and interfaces. This study focuses on exploring how auditory and vibrotactile feedback are perceived by the users when they interact in a virtual AR environment. An application was developed to the AR glasses Magic Leap with different interactions, both passive and active. An experimental study was conducted with 28 participants that got to interact in this virtual environment. The study included two parts. First the participants interacted in the virtual environment where they did a think aloud. Thereafter they were interviewed. There were a total of three test cases. One with only auditory feedback, one with vibrotactile feedback, and a third that had both auditory and vibrotactile feedback. Seven of the 28 participants acted as a control group that did not have any feedback to their interactions. The study shows that using only vibrotactile feedback creates different impressions depending on earlier experiences with the same AR environment. Using only auditory feedback created an atmosphere that were close to reality. Having both feedbacks active at the same time reduced the noticed feedback and some interactions were here not even noticed at all. Passive interactions were more noticed than active interactions in all cases. / Det finns flera olika sätt att interagera med olika hårdvaror och därför är det viktigt att ha en förståelse kring vilka faktorer som påverkar upplevelsen när man designar för diverse gränssnitt och interaktioner. Den här studien fokuserar på att utforska hur auditiv och vibrationsåterkoppling uppfattas av användaren när de interagerar i en virtuell AR-miljö. En applikation var utvecklad till AR-glasögonen Magic Leap One med olika aktiva och passiva interaktioner.En experimentell studie genomfördes med 28 deltagare som fick interagera i en virtuell miljö. Studien bestod av av två delar. Först fick deltagarna interagera i en virtuell miljö där de gjorde en think aloud. Efter detta blev de intervjuade. Det var totalt tre testfall, ett hade endast auditiv återkoppling, ett hade vibrationsåterkoppling och det sista hade både auditiv och vibrationsåterkoppling. Sju av de 28 deltagarna agerade kontrollgrupp och de hade ingen återkoppling på deras interaktioner.Studien visade att bara använda vibrationsåterkoppling skapade olika intryck beroende på de tidigare erfarenheterna i samma AR-miljö. Att endast använda auditiv återkoppling skapade en atmosfär som vara nära verkligheten. Att ha båda återkopplingarna aktiva samtidigt reducerade den totala märkta återkopplingen och några interaktioner hade inte någon person som noterade någon av dem. Passiva interaktioner var mer uppmärksammade än aktiva interaktioner i alla testfallen.

The aim of this thesis is to understand the impact of virtual reality (VR) on user's performance and experience in practical industrial applications. In particular, I collaborated with Volvo Cars supporting their current engineering design practices. VR has the potential to augment and thereby improve a person's observational analysis. Current VR publications are focused on VR applications and their potential benefits and improvements, mainly towards science, industry, medicine, and education. Yet, no research on the impact of augmenting the exploration and analysis of water contamination events in a simulated car engine has been done. In this thesis, this issue has been explored by performing a task-based controlled user study with two conditions: the Workstation and the Wall. The Workstation is the current system examining contamination events in a video with keyboard and mouse on a normal desktop screen. The Wall is the developed system examining contamination events through interaction with an Xbox-controller and Kinect head-tracking motion control on a 4K screen (400 x 240 cm projection surface) with stereoscopy using passive 3D glasses. This thesis is a pilot to discover meaningful differences and trade-offs for future studies. The result of the study shows significant differences between the two conditions. Where the Workstation is faster, but less precise, the Wall is more precise and gives more options for analysis. Nevertheless, the current version of the Wall has some weaknesses, in particular, lag and discrete steps in the rotation following the head-tracking, which meant increased duration performing the task. Overall, the high level of interaction using the Wall, the details from the 4K screen, and the stereoscopic rendering were seen as helpful in solving the task. However, the participants were not unanimous on which features were most important or whether or not every feature was essential. Therefore, research, beyond the scope of this thesis, needs to be done to examine how these individual factors affect task performance. / Syftet med detta examensarbete är att studera vilken effekt virtual reality (VR) har på användares prestation och upplevelse när de använder industriella applikationer avsedda för att utföra uppgiften i den verkliga världen. Jag samarbetade med Volvo Cars i att stödja deras nuvarande tekniska designmetod. VR har möjligheten att förstärka och därigenom förbättra analysen av en persons observationer. Nuvarande forskning rörande VR fokuserar på potentiella fördelar och möjliga förbättringar i VR applikationer, riktad mot vetenskap, industri, medicin, utbildning m.m. Emellertid har inte någon forskning gjorts över effekterna av att förstärka sättet att utforska och analysera vattenföroreningshändelser i en simulerad bilmotor. I detta examensarbete har denna fråga studerats genom en uppgiftsbaserad användarstudie med två olika metoder, nämligen Workstation och Wall. Workstation är det system som idag används för att leta efter vattenföroreningshändelser, vilket sker genom en video med tangentbord och mus på en vanlig datorskärm. Wall är ett mer utvecklat system där förekomsten av vattenföroreningshändelser kontrolleras med hjälp av en Xbox-spelkontroll och Kinect "head-tracking" rörelsekontroller på en 4K skärm (400 x 240 cm projektionsyta) med stereoskopi med 3D-glasögon. Examensarbetet är en pilotstudie för att upptäcka skillnader mellan dessa metoder i syfte att hitta infallsvinklar inför framtida studier. Resultatet av studien visar en signifikant skillnad mellan de två metoderna. Workstation är snabbare men mindre exakt medan Wall är mer noggrann och ger fler analysmöjligheter. Den nuvarande versionen av Wall har dock problem med bl.a. fördröjningar som gör att tidsåtgången för att utföra en specifik uppgift förlängs. Sammanfattningsvis ansågs interaktionen i Wall, detaljerna från 4K skärmen och 3D som hjälpsamma för att lösa uppgiften, men deltagarna var inte eniga vilken funktion som var viktigast eller huruvida varje funktion var nödvändig. Ytterligare forskning behöver därför utföras för att utröna vilka enskilda faktorer som har betydelse för deltagarnas prestation.

In Mixed Reality scenarios, background replacement is a common way to immerse a user in a synthetic environment. Properly identifying the background pixels in an image or video is a dif- ficult problem known as matting. Proper alpha mattes usually come from human guidance, special hardware setups, or color dependent algorithms. This is a consequence of the under-constrained nature of the per pixel alpha blending equation. In constant color matting, research identifies and replaces a background that is a single color, known as the chroma key color. Unfortunately, the algorithms force a controlled physical environment and favor constant, uniform lighting. More generic approaches, such as natural image matting, have made progress finding alpha matte solutions in environments with naturally occurring backgrounds. However, even for the quicker algorithms, the generation of trimaps, indicating regions of known foreground and background pixels, normally requires human interaction or offline computation. This research addresses ways to automatically solve an alpha matte for an image in realtime, and by extension a video, using a consumer level GPU. It does so even in the context of noisy environments that result in less reliable constraints than found in controlled settings. To attack these challenges, we are particularly interested in automatically generating trimaps from depth buffers for dynamic scenes so that algorithms requiring more dense constraints may be used. The resulting computation is parallelizable so that it may run on a GPU and should work for natural images as well as chroma key backgrounds. Extra input may be required, but when this occurs, commodity hardware available in most Mixed Reality setups should be able to provide the input. This allows us to provide real-time alpha mattes for Mixed Reality scenarios that take place in relatively controlled environments. As a consequence, while monochromatic backdrops (such as green screens or retro-reflective material) aid the algorithm’s accuracy, they are not an explicit requirement. iii Finally we explore a sub-image based approach to parallelize an existing hierarchical approach on high resolution imagery. We show that locality can be exploited to significantly reduce the memory and compute requirements of previously necessary when computing alpha mattes of high resolution images. We achieve this using a parallelizable scheme that is both independent of the matting algorithm and image features. Combined, these research topics provide a basis for Mixed Reality scenarios using real-time natural image matting on high definition video sources.

Imagination and fantasy environments created by writers and artists have always drawn people into their worlds. Advances in technology have blurred the lines between reality and imagination. My interest has always been to question the validity of these worlds and their cultures and to transcend the evolving virtual dimension by fusing it with what we perceive to be reality.

<p>Fluid power education would benefit from the adoption of an alternative to traditional hands-on instructional methods. Hands-on education is invaluable because it offers students experience interacting with and controlling fluid power systems and components, but systems are typically space-consuming and expensive. The study sought to prove the viability of mixed reality (MR) as an alternative to traditional hands-on fluid power instruction through the creation of MR lab exercises. A summary of design methodology was created to demonstrate how virtual fluid power components were modeled and presented in a mixed reality environment. Data was collected from students enrolled at Purdue University who participated in traditional and mixed reality fluid power lab exercises. Student responses were expected to express a positive reception of mixed reality as a fluid power instructional tool. The study anticipated that utilizing mixed reality in a fluid power laboratory setting would increase student comprehension of fluid power concepts. Educational variables were limited by restricting testing to students within the advanced fluid power course of Purdue University’s Polytechnic Institute. Students in this course provided feedback that drew comparisons between traditional and mixed reality instructional methods. Labs were created to remain within the course schedule so as not to disrupt course curriculum. Data from Likert-type surveys were analyzed from pre- and post-lab questionnaires as well as student feedback from their experience after completing each mixed reality (MR) lab. Analysis showed that MR is a viable alternative to traditional hands-on instructional methods as students showed an increase in material comprehension of both fluid power components and concepts. Students perceived MR as a beneficial instructional tool but continued to show preference towards physical interactions with components. A combination of instructional methods is recommended.</p> <p>  </p>

Mobile learning has significant potential to be very influential in further and higher education. In this research a new definition for Mobile Educational Mixed Reality Games (MEMRG) is proposed based on a mobile learning environment. A questionnaire and a quantifying scale are utilised to assist the game developers in designing a MEMRG. A ¿Conversational Framework¿ is proposed as an appropriate psycho-pedagogical approach to teaching and learning for MEMRG. This methodology is based on the theme of a ¿conversation¿ between different actors of the learning community with the objective of building the architectural framework for MEMRG. Various elements responsible for instructing and motivating learners in educational games are utilised in an instructional-motivational model. User interface design for the games incorporates an efficient navigation system that uses contextual information, and allows the players to move seamlessly between real and virtual worlds. The implementation of MEMRG using the Java 2 Micro Edition (J2ME) platform iii is presented. The hardware and software specification for the MEMRG implementation and deployment are also discussed. MEMRG has produced improvements in the different cognitive processes of the learner, and also produced a deeper level of learning through enculturation, externalising ideas, and socialising. Learners¿ enjoyment, involvement, motivation, autonomy and metacognition skills have improved. This research will assist developers and teachers to gain an insight into learning paradigms which utilise mobile game environments that are formed by mixing real and virtual spaces, and provide them with a vision for effectively incorporating these games into formal and informal classroom sessions.

In the context of Industry 4.0, the prominence of robotics has grown significantly, leading to a pressing need for advanced monitoring techniques. This thesis explores the potential role of Mixed Reality (MR) in robot condition monitoring through an exhaustive literature review of 138 selected studies. The investigation showed prevalent methods in robot condition monitoring, such as Fault Detection and Diagnosis, Machine Learning Techniques, Signal-based Monitoring, Model-based Monitoring, and Real-time Monitoring. MR, while not yet abundant in this context, is emerging as a promising tool, especially for real-time data visualization, remote maintenance, and integration with other technologies. By visually representing data and predictions directly on the robot, MR can speed up the diagnostic process, improve safety, and promote remote collaboration. However, challenges such as integration with legacy systems, effective data management, and hardware limitations were identified. The research also observed trends, benefits, and challenges in the broader application of MR in industrial settings. While MR offers significant advantages, including enhanced visualization, improved efficiency, and cost savings, its full integration into the world of robot condition monitoring necessitates further research and iterative refinement. In essence, this thesis presents a balanced overview of the potential and challenges of MR in robot condition monitoring, setting the stage for future exploration in this burgeoning domain.

Situation awareness (SA) describes how well a person perceives and understands their environment and the situation that they are in. When working in groups, shared SA describes how similarly the team members view and interpret the situation in a given environment. Immersive technology comprises technology that integrates virtual objects into the user’s reality of a physical world. It holds great potential for the application in educational contexts and collaborative settings like group projects. Immersive technology can increase engagement, make complex concepts more tangible, and increase media fluency. When immersive technology is introduced into a real-world setting, it creates a mixed reality with virtual and physical elements. In mixed reality collaborations, the complexity of elements in the environment can negatively affect the shared SA of the group members. The research problem of this thesis is that the intersection between shared SA and student group work that involves immersive technology is under-researched to this date. The research question is ”How is shared situation awareness in student group work formed when using immersive technology?”. A case study of a student group containing a participatory observation of several of their work sessions was carried out, and the obtained material was analyzed using sequential analysis. It was found that the students do not prioritize shared SA but work individually, dividing smaller subtasks among themselves and focusing on their own tasks first and foremost. Communication is used sparsely to stay updated about the other students’ work status, which helps to build shared SA. Communication also plays a crucial role in building shared SA when using immersive technology. It was also observed that the students prefer to use immersive technology in a way that allows more than one person to see the same virtual environment, as it is the case when two virtual reality (VR) headsets are connected to the same application.