The Swedish Air Force Combat Simulation Centre (FLSC) is investigating how immersive technologies can be used with flight simulators. Cybersickness is sickness-like effects experienced when immersed in virtual environments using head-mounted displays. Research suggests that cybersickness is less severe in Mixed Reality (MR), where the environment is mixed with virtual and real objects, when using an optical see-through head-mounted display. The primary objective of this thesis was to determine differences in cybersickness in a Virtual Reality (VR) and MR flight simulator when using a video see-through head-mounted display. Twenty-nine volunteers participated in a cross-over experiment with repeated measure design, at FLSC in Kista. Cybersickness was assessed using the Simulator Sickness Questionnaire (SSQ) and Fast Motion Sickness scale (FMS). Results showed that the mean total score of SSQ was higher in VR (M = 15.6) than MR (M = 12.6) although not statistically significant (p = .44). The mean FMS was also higher in VR (M = 1.34) than MR (M = 0.83) at a significant level (p = .035). Subscale scores of the SSQ showed that disorientation symptoms had a higher mean in both VR and MR, compared to nausea and oculomotor discomfort symptoms. Only in MR was disorientation symptoms significantly higher than nausea (p = .011). The thesis concluded that cybersickness seems less severe in MR and that disorientation symptoms seems predominant in MR and VR. / Flygvapnets Luftstridssimuleringscentrum (FLSC) undersöker hur immersiva teknologier kan användas vid flygsimulering. Cybersjuka är sjukdomsliknande effekter som upplevs i virtuella miljöer med huvudmonterade bildskärmar. Forskning tyder på att cybersjuka är mildare i en "blandad verklighet" (MR), där miljön är blandad med virtuella och verkliga objekt, när man använder huvudmonterade bildskärmar med optik-genomseende. Det primära syftet med detta examensarbete var att fastställa skillnader i cybersjuka mellan en helt virtuell verklighet (VR) och MR-flygsimulator när man använder en huvudmonterad bildskärm med kamera-genomseende. Tjugonio frivilliga deltog i en ö verkorsningsprövning med upprepad mätning, vid FLSC i Kista. Cybersjuka utvärderades med Simulator Sickness Questionnaire (SSQ) och Fast Motion Sickness scale (FMS). Resultaten visade att medelvärdet för totalpoängen för SSQ var högre i VR (M = 15.6) än MR (M = 12.6) men inte statistiskt signifikant (p = .44). Medelvärdet för FMS var också högre i VR (M = 1.34) än MR (M = 0.83) och statistiskt signifikant (p = .035). Subskalepoäng för SSQ visade att desorienteringssymptom hade högre medelvärden i både VR och MR, jämfört med illamående och ögonbesvär. Endast i MR var desorienteringssymptom signifikant högre än illamående (p = .011). Uppsatsen drog slutsatsen att cybersjuka verkar lindrigare i MR och att desorienteringssymptom verkar dominerande i MR och VR.

On-the-job injuries that occur even after implementing safety interventions highlight the need for identifying the limitations in them and for making future safety interventions and technological advances more effective. One possible reason for this lower-than-expected-safety returns is the latent side-effect of safety interventions, known as risk compensation. This dissertation aimed to provide empirical evidence and theoretical considerations of risk compensation effect in the construction industry. Accordingly, a multi-sensor immersive mixed-reality environment consists of a virtual projection of the environment and passive haptics of a roof was developed to study risk compensation among residential roofers. Simulating height, environmental factors (wind and sound), passive haptic, and virtual falls stimulated sufficient Sense of Presence to trigger subjects' behavioral changes while installing shingles on a 27-degree sloped roof under three levels of safety interventions (i.e., with no fall-safety intervention, with an injury-reducing fall-safety intervention—i.e., fall-arrest system—and with an injury-preventing fall-safety intervention— i.e., a fall-arrest system and a guardrail). The baseline demographic, psychographic, and cognitive measures combined with real-time tracking and wearable sensors provided an opportunity to track the worker's motions, localize his/her position, obtain real-time musculoskeletal data, and monitor the his/her behavioral and physiological responses. The collected data is then translated into information about the risk perception and risk-taking behavior of the worker. The results yielded unequivocal evidence of risk compensation—the lower perceived risk associated with the situation (lower levels of stress) and the false sense of security among roofers when they were provided with safety interventions apparently encouraged them to be less cautious by leaning over the edge, stepping closer to the roof edge, spending more time exposing themselves to fall risk, over-relying on the safety equipment through different facing directions and choices of posture stability. As a result, they also experienced more near-misses (close calls). This behavioral adaptation was more pronounced when they were provided with an injury-preventing safety intervention (e.g., guardrail). The findings also suggested that the productivity and safety benefits of safety interventions can be negated due to risk compensation, which identifies vital information for the construction-safety community to consider during the design and implementation of more effective safety interventions and technological advances. Roofers with high risk tolerance and sensation seekers were identified as high-risk groups who are more likely to be involved in risk-compensatory behaviors; various behavioral interventions are suggested in this dissertation to counteract excessive risk-taking and to reduce risk compensation. The findings of this study shed light on the question of why injury rates have remained at worrisome levels despite advances in protective measures and interventions. In the long-term, a better understanding of risk compensation will translate into fundamental knowledge about how the construction industry should approach and maintain controls after safety interventions. / Doctor of Philosophy / While researchers have dispensed considerable efforts to reduce the risk of occupational injuries by implementing safety interventions, the large number of safety incidents occurring each year in the construction industry. It is hypothesized that the latent effect of safety interventions, known as risk compensation, might be a possible reason why many of the safety interventions and technological advances have not fully achieved their safety objectives. This dissertation aimed to empirically examine the changes in workers' productivity, risk perception, risk-taking behaviors as a function of different safety interventions in place. To study this within a risk-free setting, an immersive mixed-reality environment simulating roofing task was developed. Then, the reactionary behavioral responses of participants were monitored using real-time tracking sensors and qualitative sources of data while they were completing a roofing task under three counterbalanced levels of safety interventions (i.e., with no fall-safety intervention, with an injury-reducing fall-safety intervention—i.e., fall-arrest system—and with an injury-preventing fall-safety intervention— i.e., a fall-arrest system and a guardrail). The findings indicated that the reduced perceived risk and the desire for increased productivity may skew risk analysis and strongly bias workers toward presuming invulnerability when safety interventions are in place. According to risk compensation theory, workers' risk tolerance and perceptions of risk influence their risk-taking behavior—as the perceived risk associated with the situation decreases, individuals take more risks to achieve a level of risk they can comfortably tolerate. Therefore, the workers might become less cautious by leaning over the edge, stepping closer to the roof edge, spending more time exposing themselves to fall risk, over-relying on the safety equipment through different facing directions and choices of posture stability. This result does not necessarily imply the safety innovations are completely ineffective, but rather demonstrates dangers users face when they misperceive the effectiveness of a safety intervention. Furthermore, roofers with high risk tolerance and a high sensation-seeking disposition were identified as high-risk groups who are more likely to be involved in risk-compensatory behaviors. This research represented a substantive departure from the status quo by proposing novel pathways for proactive incident prevention due to risk compensation in the construction industry. The contribution of this study is especially significant because a better understanding of risk compensation will translate into fundamental knowledge about how the construction industry should approach and maintain controls after safety interventions.

My thesis project explores a new paradigm of mood board tool that challenges the current paradigm of confining everything to 2D, flat interfaces. while investi- gating how AI can inspire and assist industrial designers during the form explora- tion process. Using a human-centered design approach, the project led to the development of COSMO, an AI-integrated, inspiration-triggering mood board tool for industrial designers. It addresses the issue of limited variety in current digital inspiration platforms, where similar content is often suggested due to AI algorithms. COSMO encourages using a variety of inspiration sources, including digital visuals and real-life experiences. This approach fosters creativity and personal connection, helping designers deeply resonate with the inspiration they collect and retrieve it at a more profound level. COSMO also utilizes Mixed Reality technology to offer an immersive, distrac- tion-free experience when organizing and internalizing inspiration sources. It offers a virtual meditative space that uses AI-assisted association loops to build associations. These loops create multiple layers of associations, assisting human designers in widening their initial associations. This leads to inspiration sources that are semantically different but insightful.

<p dir="ltr">Construction workers often suffer from excessive stress from working in dynamic and complex hazard-rich environments. These workers are subject to experiencing diverse external stressors, which can increase their involvement in risk-taking behaviors by increasing human error, referring to individuals’ misperceptions and misjudgment. Task, social, and environmental stressors are the most common external stressors that can negatively impact workers’ safety performance. Task stressors mainly occur when the projects fall behind schedule which puts workers under productivity and mental demand. In addition, workers are exposed to social stressors due to the inherently social environment of construction job sites requiring collaborative efforts. Such workers also suffer from environmental stressors as they mainly need to perform construction tasks outdoors in extreme environments. There is a paucity of research to empirically examine how such external stressors may affect workers’ situational awareness and risk-taking behaviors. Therefore, the overall goal of this dissertation is to <i>examine the theoretical foundations and empirical evidence of changes in workers’ decision dynamic in the construction industry when exposed to task (e.g., productivity pressure and mental demand), social (e.g., peer pressure), and environmental (e.g., heat stress) stressors.</i></p><p dir="ltr">To accomplish this, a series of studies were conducted to investigate the effects of task, social, and environmental stressors on workers’ situational awareness and hazard identification skills. To do so, taking advantage of novel technologies, this study developed immersive mixed reality (MR) and augmented virtuality (AV) simulating high-risk construction tasks. Such environments were integrated with several wearable sensing technologies to measure individuals’ cognitive responses and decision dynamics while completing the tasks under different stressors. The findings demonstrated that external stressors reduce worker situational awareness, impair their cognitive processes, and negatively affect their safety performance.</p><p dir="ltr">Such findings were then utilized to develop an intelligent and comprehensive AI-based predictive system to identify at-risk workers imposed to external stressors. This system translates physiological, cognitive, and biomechanical metrics into AI-identified predictors of three types of external stressors; localizes workers, and assesses risks of being injured in real-time which will then dictate the urgency of providing any intervention. These analyses are then used to identify and propose tailored safety interventions.</p><p dir="ltr">This dissertation contributes to the existing body of knowledge by adopting innovative approaches to empirically study the extent to which external stressors may affect workers’ decision dynamics by examining the changes in their situational awareness, risk-taking, and safety performance measures. In addition, this work contributes to practice by raising awareness about the adverse effects of several cognitive biases due to such stressors, such as risk compensation, cognitive tunneling, and impaired attentional distribution, which can undermine the efficacy of safety interventions in the construction industry. It highlights the critical role of these cognitive biases in safety practices and the necessity of educating safety professionals and workers about how psychological factors can impact safety on the job site and potential ways to mitigate these potential negative impacts. Further, the developed AI-based predictive system breaks new ground by identifying at-risk workers, assessing potential risks, and recommending safety interventions.</p>

Tato práce se zaměřuje na použitelnost brýlí Microsoft HoloLens pro rozšířenou realitu v prototypu pracoviště pro spolupráci člověka s robotem - "ARTable". Použití brýlí je demonstrováno vytvořeným uživatelským rozhraním, které pomáhá uživatelům lépe a rychleji porozumět systému ARTable. Umožňuje prostorově vizualizovat naučené programy, aniž by bylo nutné spouštět samotného robota. Uživatel je veden 3D animací jednotlivých programů a hlasem zařízení, což mu pomůže získat jasnou představu o tom, co by se stalo, pokud by program spustil přímo na robotovi. Implementované řešení také umožňuje interaktivně provést uživatele celým procesem programování robota. Použití brýlí umožňuje mimo jiné zobrazit cenné prostorové informace, například vidění robota, tedy zvýraznit ty objekty, které jsou robotem detekovány.

Digital environmental simulations can be performed with different techniques and the most common technologies are virtual reality, augmented reality and mixed reality. Digital environmental simulations have proven to be effective in practicing surgery, industrial activities and for military exercises. Previous studies have shown that technology habits are a factor that affects whether digital environmental simulations can be used effectively. Thus, the purpose of the study was to investigate how users can be introduced to digital environmental simulations. To achieve the purpose, the following question was needed: How can a digital prototype be designed to introduce users to digital environmental simulations based on user needs? The study was based on design science as a research strategy, which meant that the study was carried out in three phases: development of requirements, development and evaluation of digital prototype. The production of requirements was made through a qualitative data collection in the form of semi-structured interviews. The interview questions were developed using a theoretical framework on digital competence. The interviews resulted in a requirement specification containing 15 user stories that were prioritized.Based onthe requirement specification, a digital prototype was developed in thedevelopment environment Unity. The evaluation of the digital prototype wascarried out in two stages, where the first was to evaluate internally and thesecond step was to evaluate externally. The external evaluation was conductedwith respondents who carried out a use test of the digital prototype thatresulted in proposals for further development. But it also resulted in usershaving increased knowledge and ability to see opportunities with digitalenvironmental simulations. The conclusion is that users can be introduced to digitalenvironmental simulations through a digital prototype designed based on userneeds. / Digitala miljösimuleringar kan utföras med olika tekniker och de vanligaste teknikerna är virtual reality, augmented reality och mixed reality. Digitala miljösimuleringar har visat sig vara effektiva för att öva på kirurgi, industrimoment samt för militärövningar. Tidigare studier har visat att teknikvana är en faktor som påverkar om digitala miljösimuleringar kan användas effektivt. Således var syftet med studien att undersöka hur användare kan introduceras till digitala miljösimuleringar. För att uppnå syftet behövdes följande frågeställning besvaras: Hur kan en digital prototyp utformas för att introducera användare till digitala miljösimuleringar baserat på användares behov? Studien har utgått från design science som forskningsstrategi, vilket medförde att studien har utförts i tre faser: framtagning av krav, utveckling och utvärdering av digital prototyp. Framtagning av krav skedde genom en kvalitativ datainsamling i form av semistrukturerade intervjuer. Intervjufrågorna togs fram med hjälp av ett teoretiskt ramverk om digital kompetens. Intervjuerna resulterade i en kravspecifikation innehållande 15 användarberättelser som prioriterades.   Utifrån kravspecifikationen utvecklades en digital prototyp i utvecklingsmiljön Unity. Utvärderingen av den digitala prototypen genomfördes i två steg, där det första var att utvärdera internt och det andra steget var att utvärdera externt. Den externa utvärderingen genomfördes med respondenter som utförde ett användningstest av den digitala prototypen som resulterade i förslag till vidareutvecklingMen det resulterade även i att användare fick ökadkunskap och förmåga att se möjligheter med digitala miljösimuleringar.Slutsatsen är att användare kan introduceras till digitala miljösimuleringargenom en digital prototyp som utformats baserat på användares behov.

Approved for public release, distribution is unlimited / Once the military helicopter pilot deploys aboard a naval vessel he leaves behind all training platforms, short of the actual aircraft, that present enough fidelity for him to maintain the highest levels of readiness. To that end, this thesis takes a preliminary step in creating a trainer that places the pilot in an immersive and familiar environment to exercise myriad piloting tasks as faithfully and as rigorously as in actual flight. The focus of this thesis it to assess the viability of an chromakeyed augmented virtual environment (ChrAVE) trainer embedded into a helicopter for use in maintaining certain perishable skills. Specifically this thesis will address the task of helicopter low-level land navigation. The ChrAVE was developed to substantiate the viability of having embedded trainers in helicopters. The ChrAVE is comprised of commercial off the shelf (COTS) equipment on a transportable cart. In determining whether a system such as the ChrAVE is viable as a laboratory for continued training in virtual environment, the opinion of actual pilots that were tasked with realistic workloads was used. Additionally, empirical data was collected and evaluated according to the subject pool's thresholds for acceptable low-level navigation performance. / Captain, United States Marine Corps

We propose the concept of spatial analytic interfaces (SAIs) as a tool for performing in-situ, everyday analytic tasks. Mobile computing is now ubiquitous and provides access to information at nearly any time or place. However, current mobile interfaces do not easily enable the type of sophisticated analytic tasks that are now well-supported by desktop computers. Conversely, desktop computers, with large available screen space to view multiple data visualizations, are not always available at the ideal time and place for a particular task. Spatial user interfaces, leveraging state-of-the-art miniature and wearable technologies, can potentially provide intuitive computer interfaces to deal with the complexity needed to support everyday analytic tasks. These interfaces can be implemented with versatile form factors that provide mobility for doing such taskwork in-situ, that is, at the ideal time and place. We explore the design of spatial analytic interfaces for in-situ analytic tasks, that leverage the benefits of an upcoming generation of light-weight, see-through, head-worn displays. We propose how such a platform can meet the five primary design requirements for personal visual analytics: mobility, integration, interpretation, multiple views and interactivity. We begin with a design framework for spatial analytic interfaces based on a survey of existing designs of spatial user interfaces. We then explore how to best meet these requirements through a series of design concepts, user studies and prototype implementations. Our result is a holistic exploration of the spatial analytic concept on a head-worn display platform. / October 2016

Integrating state-of-the-art sensory and display technologies with 3D computer graphics, extended reality (XR) affords capabilities to create enhanced human experiences by merging virtual elements with the real world. To better understand how Sound and Music Computing (SMC) can benefit from the capabilities of XR, this thesis presents novel research on the de- sign of musical experiences in extended reality (MusE-XR). Integrating XR with research on computer assisted musical instrument tutoring (CAMIT) as well as New Interfaces for Musical Expression (NIME), I explore the MusE-XR design space to contribute to a better understanding of the capabilities of XR for SMC. The first area of focus in this thesis is the application of XR technologies to CAMIT enabling extended reality enhanced musical instrument learning (XREMIL). A common approach in CAMIT is the automatic assessment of musical performance. Generally, these systems focus on the aural quality of the performance, but emerging XR related sensory technologies afford the development of systems to assess playing technique. Employing these technologies, the first contribution in this thesis is a CAMIT system for the automatic assessment of pianist hand posture using depth data. Hand posture assessment is performed through an applied computer vision (CV) and machine learning (ML) pipeline to classify a pianist’s hands captured by a depth camera into one of three posture classes. Assessment results from the system are intended to be integrated into a CAMIT interface to deliver feedback to students regarding their hand posture. One method to present the feedback is through real-time visual feedback (RTVF) displayed on a standard 2D computer display, but this method is limited by a need for the student to constantly shift focus between the instrument and the display. XR affords new methods to potentially address this limitation through capabilities to directly augment a musical instrument with RTVF by overlaying 3D virtual objects on the instrument. Due to limited research evaluating effectiveness of this approach, it is unclear how the added cognitive demands of RTVF in virtual environments (VEs) affect the learning process. To fill this gap, the second major contribution of this thesis is the first known user study evaluating the effectiveness of XREMIL. Results of the study show that an XR environment with RTVF improves participant performance during training, but may lead to decreased improvement after the training. On the other hand,interviews with participants indicate that the XR environment increased their confidence leading them to feel more engaged during training. In addition to enhancing CAMIT, the second area of focus in this thesis is the application of XR to NIME enabling virtual environments for musical expression (VEME). Development of VEME requires a workflow that integrates XR development tools with existing sound design tools. This presents numerous technical challenges, especially to novice XR developers. To simplify this process and facilitate VEME development, the third major contribution of this thesis is an open source toolkit, called OSC-XR. OSC-XR makes VEME development more accessible by providing developers with readily available Open Sound Control (OSC) virtual controllers. I present three new VEMEs, developed with OSC-XR, to identify affordances and guidelines for VEME design. The insights gained through these studies exploring the application of XR to musical learning and performance, lead to new affordances and guidelines for the design of effective and engaging MusE-XR. / Graduate

Les Learning Games sont des environnements d’apprentissage, souvent informatisés, qui utilisent des ressorts ludiques pour catalyser l’attention des apprenants et ainsi faciliter leur apprentissage. Ils ont des atouts indéniables mais présentent également certaines limites, comme des situations d’apprentissage trop artificielles. Ces limites peuvent être dépassées par l’intégration d’interactions en Réalité Mixte dans les Learning Games, que nous appelons alors des Mixed Reality Learning Games (MRLG). La Réalité Mixte, qui combine environnements numériques et objets réels, ouvre de nouvelles possibilités d’interactions et d’apprentissage qui gomment les limites précédentes et qu’il faut repérer et explorer. Dans ce contexte, nous nous intéressons au processus de conception des MRLG. Dans un premier temps, nous présentons une étude sur l’utilisation de la Réalité Mixte dans les domaines de l’apprentissage et du jeu, incluant un état de l’art des MRLG. Cette étude montre que, malgré de nombreux atouts, la conception des MRLG reste difficile à maîtriser. En effet, il n’existe ni méthode ni outil adapté à la conception de ce type d’environnements. Dans un second temps, nous analysons et modélisons l’activité de conception des MRLG à travers la littérature et des expériences de conception, dont une menée dans le cadre du projet SEGAREM. Cette démarche révèle des verrous spécifiques tels que l’absence d’aide à la modélisation (ou formalisation), à la créativité et à la vérification de la cohérence des idées. Nous éclairons nos réponses à ces besoins par un recensement des outils utilisés dans les domaines liés aux MRLG : situations d’apprentissage, jeux et environnements de la Réalité Mixte. Ceci nous amène à proposer deux outils conceptuels : un modèle de description de MRLG (f-MRLG) et des aides à la créativité sous la forme de propositions puis de recommandations. Le modèle de description a pour objectif de formaliser l’ensemble des éléments constituant un MRLG, mais aussi d’être un moyen d’identifier les éléments à définir, de structurer et de vérifier les idées. Les listes de propositions et recommandations ont pour but d’aider le concepteur à faire des choix cohérents par rapport à la situation d’apprentissage visée, en particulier en ce qui concerne les types de jeux et les dispositifs de Réalité Mixte. Une première évaluation de ces propositions a conduit à leur amélioration. Ces propositions sont à l’origine de la conception et du développement d’un outil auteur informatisé : MIRLEGADEE (Mixed Reality Learning Game DEsign Environment). MIRLEGADEE est basé sur LEGADEE, un environnement auteur pour la conception de Learning Games. Une expérimentation auprès de 20 enseignants et concepteurs de formation a validé le bienfondé de cet outil qui guide effectivement les concepteurs dans les phases amont du processus de conception de MRLG malgré des limites pour l’accompagnement de tâches complexes. / Game-based learning is one efficient pedagogical concept that uses game principles to incite learners to engage into learning activities. Learning Games (LG) are commonly known as digital environments. They have undeniable assets but also some limits, such as the artificiality of the learning context. In the mean time, new technologies have been increasingly developed, thus providing new perspectives in game-based learning. In particular, Mixed Reality (MR) technologies merge both real and digital worlds. Mixed Reality Learning Games (MRLG) offer real benefits for teaching: they enable active pedagogy trough the physical immersion of learners, “in situ” information while practicing and authentic context. In our work, we focus on the design process of MRLG. The first part of the thesis presents how Mixed Reality is used for educational and gaming purposes. An analysis of existing MRLG shows both their assets and the complexity of their design. MRLG designers have to cope with all the difficulties of learning design, game design and mixed reality design at the same time, and with the integration of all aspects in a coherent way. Besides, there is a lack of specific tool or methodology. In order to understand the specific needs of MRLG designers, we analyze and model the MRLG design activity from MRLG design processes described in papers and existing methodologies for LG and MR. We also illustrate and clarify MRLG design needs with the observation of a MRLG design activity in the SEGAREM project. We highlight some needs for modeling, creativity, and verification of coherence. To meet the identified needs, the third part is dedicated to a state of the art of tools available for learning design, game design and Mixed Reality design. This study leads us to three solutions to assist MRLG design: a model, a set of tools for creativity, and an authoring tool. We first propose a model called f-MRLG to describe fully and clearly a MRLG. f-MRLG is a support to MRLG design as it helps designer to organize their ideas and to identify which elements must be described. It also reinforces mutual comprehension in a team. Our second proposal is a set of tools for creativity: lists of possibilities, examples and suggestions for game types and Mixed Reality systems choices. We conducted a first experiment on the two proposals, which led to their improvement. These two proposals drove to the design and development of an authoring tool, named MIRLEGADEE (MIxed Reality LEarning GAme DEsign Environment), to support the MRLG design. This tool is an extension of LEGADEE, which already supports the design of learning games using a computer. An experiment with 20 teachers and training designers validated that MIRLEGADEE successfully guides the designers in the MRLG design process, in spite of limits for the support of complex tasks.