With the help of XR tools, globally dispersed teams can collaborate remotely in shared virtual environments, reducing the costs associated with physical prototypes and travel while benefiting from including stakeholders from various backgrounds in their process. Integrating digital human models in these virtual environments allows for collaborative design interactions and possibilities for performing ergonomic design and assessments. While screen-based tools are the standard for these evaluations, the inclusion of XR tools can augment current screen-based tools to improve the evaluation of ergonomics and the assembleability of components with digital human models. This study describes an implementation model for collaborative assembly simulations, ergonomics assessments, and reviews in a shared virtual environment with XR and screen-based tools. A pilot study with fifteen participants was designed to compare a Swedish/Chinese vehicle manufacturer's approach to simulating and analyzing assembly designs with the collaborative virtual environment model developed. The results show that teams collaborating in a shared virtual environment performed better in user experience, the feeling of presence, and precision when detecting ergonomics and assemblability issues.

Integrating state-of-the-art sensory and display technologies with 3D computer graphics, extended reality (XR) affords capabilities to create enhanced human experiences by merging virtual elements with the real world. To better understand how Sound and Music Computing (SMC) can benefit from the capabilities of XR, this thesis presents novel research on the de- sign of musical experiences in extended reality (MusE-XR). Integrating XR with research on computer assisted musical instrument tutoring (CAMIT) as well as New Interfaces for Musical Expression (NIME), I explore the MusE-XR design space to contribute to a better understanding of the capabilities of XR for SMC. The first area of focus in this thesis is the application of XR technologies to CAMIT enabling extended reality enhanced musical instrument learning (XREMIL). A common approach in CAMIT is the automatic assessment of musical performance. Generally, these systems focus on the aural quality of the performance, but emerging XR related sensory technologies afford the development of systems to assess playing technique. Employing these technologies, the first contribution in this thesis is a CAMIT system for the automatic assessment of pianist hand posture using depth data. Hand posture assessment is performed through an applied computer vision (CV) and machine learning (ML) pipeline to classify a pianist’s hands captured by a depth camera into one of three posture classes. Assessment results from the system are intended to be integrated into a CAMIT interface to deliver feedback to students regarding their hand posture. One method to present the feedback is through real-time visual feedback (RTVF) displayed on a standard 2D computer display, but this method is limited by a need for the student to constantly shift focus between the instrument and the display. XR affords new methods to potentially address this limitation through capabilities to directly augment a musical instrument with RTVF by overlaying 3D virtual objects on the instrument. Due to limited research evaluating effectiveness of this approach, it is unclear how the added cognitive demands of RTVF in virtual environments (VEs) affect the learning process. To fill this gap, the second major contribution of this thesis is the first known user study evaluating the effectiveness of XREMIL. Results of the study show that an XR environment with RTVF improves participant performance during training, but may lead to decreased improvement after the training. On the other hand,interviews with participants indicate that the XR environment increased their confidence leading them to feel more engaged during training. In addition to enhancing CAMIT, the second area of focus in this thesis is the application of XR to NIME enabling virtual environments for musical expression (VEME). Development of VEME requires a workflow that integrates XR development tools with existing sound design tools. This presents numerous technical challenges, especially to novice XR developers. To simplify this process and facilitate VEME development, the third major contribution of this thesis is an open source toolkit, called OSC-XR. OSC-XR makes VEME development more accessible by providing developers with readily available Open Sound Control (OSC) virtual controllers. I present three new VEMEs, developed with OSC-XR, to identify affordances and guidelines for VEME design. The insights gained through these studies exploring the application of XR to musical learning and performance, lead to new affordances and guidelines for the design of effective and engaging MusE-XR. / Graduate

With a lineage originating in the days of silent films, visual music, in its current incarnation, is a relatively recent phenomenon when compared to an historically broad field of creative expression. Today it is a time-based audio/visual territory explored and mined by a handful of visual and musical artists. However, an extensive examination of the literature indicates that few of these composers have delved into the associable areas of merging virtual holography and holophony toward visual music composition. It is posited here that such an approach is extremely rich with novel expressive potential and simultaneously with numerous novel challenges. The goal of this study is, through praxis, to instantiate and document an initial exploration into the implementation of holochory toward the creation of visual music compositions. Obviously, engaging holochoric visual music as a means of artistic expression requires an interdisciplinary pipeline. Certainly, this is demonstrated in merging music and visual art into a cohesive form, which is the basis of visual music composition. However, in this study is revealed another form of interdisciplinarity. A major challenge resides with the development of the means to efficiently render the high-resolution stereoscopic images intrinsic to the animation of virtual holograms. Though rendering is a challenge consistent with creating digital animations in general, here the challenge is further exacerbated by the extensive use of multiple reflections and refractions to create complexity from relatively simple geometric objects. This reveals that, with the level of computational technology currently available, the implementation of high-performance computing is the optimal approach. Unifying such diverse areas as music, visual art, and computer science toward a common artistic medium necessitates a methodological approach in which the interdependency between each facet is recognized and engaged. Ultimately, a quadrilateral reciprocative feedback loop, involving the composer's sensibilities in addition to each of the other facets of the compositional process, must be realized in order to facilitate a cohesive methodology leading toward viability. This dissertation provides documentation of methodologies and ideologies undertaken in an initial foray into creating holochoric visual music compositions. Interlaced matrices of contextualization are intended to disseminate the processes involved in deference to composers who will inevitably follow in the wake of this research. Accomplishing such a goal is a quintessential aspect of practice-based research, through which new knowledge is gained during the act of creating. Rather than formulating theoretical perspectives, it is through the praxis of composing holochoric visual music that the constantly arising challenges are recognized, analyzed, and subsequently addressed and resolved in order to ensure progression in the compositional process. Though measuring the success of the resultant compositions is indeed a subjective endeavor, as is the case with all art, the means by which they are achieved is not. The development of such pipelines and processes, and their implementation in practice, are the basic building blocks of further exploration, discovery, and artistic expression. This is the impetus for this document and for my constantly evolving and progressing trajectory as a scholar, artist, composer, and computer scientist. / Doctor of Philosophy / In this paper the author explores the idea that, owing to their shared three-dimensional nature, holophons and holograms are well suited as mediums for visual music composition. This union is ripe with creative opportunity and fraught with challenges in the areas of aesthetics and technical implementation. Squarely situated upon the bleeding edge of phenomenological research and creative practice, this novel medium is nonetheless within reach. Here, one methodological pipeline is delineated that employs the convergence of holography, holophony, and super-computing toward the creation of visual music compositions intended for head mounted displays or large scale 3D/360 projection screens and high-density loudspeaker arrays.

Holographic communications technologies are a new digital technology being currently  developed by companies to be available to the public in the near future. Through holographic  communications technologies, the image of one user located in one place can be captured with  special equipment from different angles, so that its three-dimensional depiction is created  realistically. The 3D depiction is then transmitted in high-quality to another user located in some  other remote place, creating a whole new experience of unconventional data communication. This  new digital technology is planned to be deployed in several fields, such as marketing, education,  medicine, business and entertainment. However, like any new technological tool that is  introduced, also in the case of holographic communication technologies there are ethical and/or  legal challenges that could emerge from its misuse and set in danger fundamental human rights.  Hence, research was conducted as an interpretive qualitative study to explore the potential ethical  and/or legal challenges of holographic communications technologies by considering the  knowledge, opinions, experiences, and perspectives of legal experts, as well as experts on ethical  issues. The empirical data was collected through individual semi-structured interviews from the  selected participants, and from the review of mainly confidential documents, before being  subjected to a thematic analysis. The analysis of the data gave prominence to five themes which  were reviewed in the context of the research question and the proposed theoretical framework  with the help of Ethical Technology Assessment (ETA). The discussion of the findings revealed  that the most prominent challenges that could arise from the use of holographic communications  technologies are privacy and data protection issues. It was further shown that existing digital  technologies with similar features and the ethical and/or legal challenges confronted in the past  about them can be of help to approach the ones regarding holographic communications  technologies. The research indicated that the designing phase of a new digital technology, and  accordingly holographic communication technologies, is important as decisions are being made  about the technical features of it, as well as the capabilities it will offer to the end users. Hence, it  plays a vital role to whether the use of the digital technology will be according to ethics and laws  later on. It was also shown that the participation of end users in this phase is important to reflect  their concerns and desires with the creators before the new digital technology is introduced, but  it is still a challenge to decide who will be included and on what criteria they will be selected.   The research showed that existing ethical and legal frameworks are important to form a  corresponding framework for holographic communications technologies and can be built upon.  However, this is also a challenge, as, according to the research, any new framework that will be  constructed with regard to holographic communications technologies cannot possibly be  unanimous as different norms and societal beliefs apply to different territories or domains, but  should rather be “personalized” according to the targeted audience.

Immersion based technologies such as Augmented Reality (AR), Virtual Reality (VR) and Mixed Reality (MR), together falling under the umbrella of Extended Reality (XR) have taken the world by storm in the recent past. However, with the growing market and the increasing number of applications of XR, multiple challenges have arisen. To maintain acceptable levels of motion-to-photon latency, there is a need to serve the users with ultra low latency and with high reliability. To provide high quality rendering, these solutions have traditionally been deployed with wired connections, but severely inhibiting user mobility. Thus, the need to develop wireless solutions promising ultra low latency and high reliability emerges. Cloud/Edge based solutions promise to provide great dividends in this regard but it still remains crucial to understand how different scheduling policies perform against one another in terms of average throughput, mean system time, the number of UEs which can be serviced simultaneously etc. In this thesis, we explore how online packet scheduling policies such as first-come-first-serve, earliestdeadline-first, maximum weight scheduling etc. compare against other Quality of Experience(QoE)/ packet weight aware online scheduling policies and also against optimal offline schemes such as maximum-weighted-bipartitematching. We perform a detailed analysis of how these policies fare by studying various metrics such as the average-packet system time, competitive ratios, packet drop percentages and weight throughput, amongst others. Finally, we also explore how the introduction of multi-layered video encoding impacts XR service. Amongst the findings of the thesis, we conclude that it is possible to come up with solutions such as EDFα (which is a deadline and weight aware derivative of the earliest deadline first scheduling policy), which can either increase the weight throughput when compared to other baselines while also providing lesser packet drops and lower average system times for the scheduled packets. This algorithm can be further tuned by varying α to accordingly alter the weight throughput, system time and packet drop ratio depending on the precise user application. Additionally, we establish with the help of simulations that the introduction of multi-layered video encoding conclusively helps in reducing the average system time and eventually allows for more users to be accommodated in an XR based system at the cost of worsening video quality. / Immersionsbaserade teknologier som Augmented Reality (AR), Virtual Reality (VR) och Mixed Reality (MR), som tillsammans faller under paraplyet Extended Reality (XR) har tagit världen med storm på senare tid. Men med den växande marknaden och det ökande antalet tillämpningar av XR har flera utmaningar uppstått. För att förhindra åksjuka hos användare och för att upprätthålla acceptabla nivåer av rörelse-till-foton-latens, finns det ett behov av att betjäna användarna med ultralåg latens och med hög tillförlitlighet. För att ge högkvalitativ rendering har dessa lösningar traditionellt implementerats med trådbundna anslutningar, men de hämmar kraftigt användarens rörlighet. Därför uppstår behovet av att utveckla trådlösa lösningar som lovar ultralåg latens och hög tillförlitlighet. Moln/Edge-baserade lösningar lovar att ge stor utdelning i detta avseende, men det är fortfarande viktigt att förstå hur olika schemaläggningspolicyer fungerar mot varandra när det gäller genomsnittlig genomströmning, genomsnittlig systemtid, antalet UE:er som kan betjänas samtidigt etc. I den här avhandlingen undersöker vi hur online-paketschemaläggningspolicyer som round robin, först till kvarnförst-kvarn, tidigast-deadline-först, schemaläggning för maximal vikt etc. jämförs med andra Quality of Experience (QoE)/Viktmedvetna onlineschemaläggningspolicyer och även mot optimala offline-scheman såsom maximalt viktad-bipartite-matchning. Vi utför en detaljerad analys av hur dessa policyer klarar sig genom att studera olika mätvärden, såsom den genomsnittliga paketets systemtid, konkurrensförhållanden, procentsatser för paketnedgång och viktad genomströmning, bland annat. Slutligen undersöker vi också hur introduktionen av flerskiktad videokodning påverkar XRtjänsten. Bland resultaten av avhandlingen drar vi slutsatsen att det är möjligt att komma med lösningar som EDFα (som är en deadline- och viktmedveten derivata av Earliest deadline first scheduling policy), som antingen kan öka den viktade genomströmning jämfört med andra baslinjer samtidigt som det ger mindre paketnedgångar och lägre genomsnittliga systemtider för de schemalagda paketen. Denna algoritm kan ställas in ytterligare genom att variera α för att följaktligen ändra den viktade genomströmningen, systemtiden och paketnedgångshastigheten beroende på den exakta användarapplikationen. Dessutom fastställer vi med hjälp av simuleringar att införandet av flerskiktsvideokodning definitivt hjälper till att minska den genomsnittliga systemtiden och så småningom tillåter fler användare att få plats i ett XR-baserat system.