With the development of technology over the past decade and the former challenges of virtual environments mitigated, the need for further study of human interaction with these environments has grown apparent. The visual and interaction components for virtual reality applications have been comprehensively studied, but a lack of spatial audio fidelity leaves a need for understanding how well humans can localize aural cues and discern audio-visual disparity in these virtual environments. In order for development of accurate and efficient levels of audio fidelity, a human study was conducted with 18 participants to see how far a bimodal audio and visual cue need to separate for someone to notice. As suspected, having a visual component paired with an auditory one led to biasing toward the visual component. The average participant noticed a disparity when the audio component was 33.7° apart from the visual one, pertaining to the azimuth. There was no significant evidence to suggest that speed or direction of audio component disparity led to better localization performance by participant. Presence and prior experience did not have an effect on localization performance; however, a larger participant base may be needed to draw further conclusions. Increase in localization ability was observed within a few practice rounds for participants. Overall, performance in virtual reality was parallel to augmented reality performance when a visual source biased sound localization, and can this be a both tool and design constraint for virtual environment developers. / Master of Science / Virtual Reality has overcame a large technological gap over the past decade, allowing itself to be a strong tool in many applications from training to entertainment. The need for studying audio fidelity in virtual environments has emerged from a gap of research in the virtual reality domain. Therefore, a human participant study was conducted to see how well they could localize sound in a virtual environment. This involved signaling when they noticed a visual object and a sound split. The average of 72 trials with 18 participants was 33.7° of separation on the horizontal plane. This can be both a tool and a design constraint for virtual reality developers, who can use this visual biasing as a guideline for future applications.

This study investigated automobile drivers' ability to localize auditory and haptic (touch) alarms in terms of speed and accuracy. Thirty-two subjects, balanced across age (20-30 years old and 60-70 years old) and gender, participated in the study. Subjects were screened for minimum hearing of 40 dB for 500 Hz through 4000 Hz auditory tones, and maximum bilateral hearing differences of 10 dB. The experiment consisted of subjects identifying the target location of an alarm while driving a 2001 Buick LeSabre at 55 mph in light traffic. Four alarm modes were tested: 1) an auditory broadband alarm, 2) a haptic seat, 3) a combination of the haptic and the auditory alarm modes, and 4) a combination of the haptic alarm mode with a non-directional auditory alarm played from the front speakers of the vehicle. The alarms were evoked from eight target locations: the front-left, front, front-right, right, back-right, back, back-left, and left. The target locations of the auditory alarm mode existed around the interior of the car cabin using the vehicle's stock sound system speakers. The haptic alarm target locations existed in the bottom of the driver seat using an eight-by-eight grid of actuators. The experimenter evoked the alarms while subjects drove along a two-lane highway, and the alarms were not associated with any actual collision threat. Subjects were instructed to quickly identify the location of the alarm by calling them out, while being as correct as possible. Their choice response time and target location selection was recorded. The alarms were presented approximately every minute during fifteen-minute intervals over the duration of two and a half hours. Subjects completed questionnaires regarding their preference to the alarm modes. Under the conditions investigated, subjects localized the haptic alarm mode faster and more accurately than the auditory alarm mode. Subjects performed equally well with the haptic alarm mode and the two auditory and haptic combination alarm modes in terms of speed and accuracy in identifying their location. Subjects did express a preference for the addition of the auditory component to the haptic alarm mode, perhaps owing to a heightened sense of urgency. However, subjects preferred the haptic alarm mode on its own in response to hypothetical false alarm questions, perhaps because it was less annoying. Alarm mode discriminability was believed to affect localization accuracy and response time owing to its effect on the likelihood of correctly identifying a target location and the attention resources required to differentiate adjacent target locations. / Master of Science

Thesis (MScEng (Electrical and Electronic Engineering))--University of Stellenbosch, 2010. / ENGLISH ABSTRACT: Telephony has developed substantially over the years, but the fundamental auditory model of mixing all the audio from di erent sources together into a single monaural stream has not changed since the telephone was rst invented. Monaural audio is very di cult to follow in a multiple-source situation such as a conference call. Sound originating from a speci c point in space will travel along a slightly di erent path to each ear. Although we are not consciously aware of it, our brain processes these spatial cues to help us to locate sounds in space. It is this spatial information that allows us to focus our attention and listen to a single speaker in an environment where many di erent sources may be active at the same time; a phenomenon known as the \cocktail party e ect". It is possible to reproduce these spatial cues in a sound recording, using Head-Related Transfer Functions (HRTFs) to allow a listener to experience localised audio, even when sound is reproduced through a headset. In this thesis, spatial audio is implemented in a telephony application as well as in a virtual world. Experiments were conducted which demonstrated that spatial audio increases the intelligibility of speech in a multiple-source environment and aids active speaker identi cation. Resource usage measurements show that these bene ts are, however, not without a cost. In conclusion, spatial audio was shown to be an improvement over the monaural audio model traditionally implemented in telephony. / AFRIKAANSE OPSOMMING: Telefonie het ansienlik ontwikkel oor die jare, maar die basiese ouditiewe model waarin die klank van alle verskillende bronne bymekaar gemeng word na een enkelouditoriese stroom het nie verander sedert die eerste telefoon gebou is nie. Enkelouditoriese klank is baie moeilik om te volg in 'n meervoudigebron situasie, soos byvoorbeeld in 'n konferensie oproep. Klank met oorsprong by 'n sekere punt in die ruimte sal 'n e ens anderse pad na elke oor volg. Selfs is ons nie aktief bewus hiervan nie, verwerk ons brein hierdie ruimtelike aanduidinge om ons te help om klanke in die ruimte te vind. Dit is hierdie ruimtelike inligting wat ons toelaat om ons aandag te vestig en te luister na 'n enkele spreker in 'n omgewing waar baie verskillende bronne terselfdertyd aktief mag wees, 'n verskynsel wat bekend staan as die \skemerkelkiepartytjiee ek". Dit is moontlik om hierdie ruimtelike leidrade na 'n klank te reproduseer met behulp van hoofverwandeoordragfunksies (HRTFs) en om daardeur 'n luisteraar gelokaliseerde klank te laat ervaar, selfs wanneer die klank deur middel van oorfone gespeel word. In hierdie tesis word ruimtelike klank ge mplementeer in 'n telefonieprogram, sowel as in 'n virtuelew^ereld. Eksperimente is uitgevoer wat getoon het dat ruimtelike klank die verstaanbaarheid van spraak in 'n meerderebronomgewing verhoog en help met aktiewe spreker identi kasie. Hulpbrongebruiks metings toon aan dat hierdie voordele egter nie sonder 'n koste kom nie. Ter afsluiting, dit is bewys dat ruimtelike klank 'n verbetering tewees gebring het oor die enkelouditorieseklankmodel wat tradisioneel in telefonie gebruik het.

Nos últimos anos os avanços em áudio multicanal e sistemas envolventes despertaram um grande interesse pelas técnicas para a auralização de campos sonoros espaciais, capazes de recriar ambientações acústicas com grande realismo, envolvendo o ouvinte completamente. Um estudo sobre as tecnologias e soluções para áudio espacial mostrou que a construção de auralizadores envolve quatro blocos funcionais fundamentais. Mas, embora existam muitas técnicas e ferramentas disponíveis, não há uma arquitetura integradora para produção de áudio espacial que permita ao desenvolvedor selecionar as técnicas que deseja e montar uma solução com um grau de refinamento arbitrário que atenda a seus requisitos. Apresenta-se uma arquitetura modular aberta em quatro camadas para produção de áudio espacial em ambientes audiovisuais imersivos. Como estudo de caso, aborda-se a implementação de um auralizador Ambisonics para a CAVERNA Digital. Verifica-se que a auralização pode fornecer a correta perspectiva acústica do ambiente virtual necessária para uma percepção mais realista do espaço, e que a abordagem aberta para se implementar sistemas de áudio 2D/3D apresenta vantagens. Os resultados da implementação são apresentados e discutidos. Como conclusões, identificam-se desafios na implementação e os trabalhos futuros. / Recent advances in multichannel audio and surround systems have contributed to an increasing interest for spatial sound field auralization, capable of delivering acoustic ambience with great realism, and surrounding listener completely. A study on technologies and solutions for spatial sound has shown that building auralizators involve four major functional blocks, but, although there are many techniques and tools available, there is not an integration architecture for spatial audio production which permits developers to select their favorite techniques, and build a solution with an arbitrary refinement level, as wished. It is presented a four level modular open architecture for spatial audio production in immersive audiovisual environments. As a case study, an Ambisonics auralizator is implemented for the CAVERNA Digital CAVE. It is verified that auralization can deliver the correct acoustical perspective for the virtual environment necessary for a more realistic perception of space, and that an open approach presents advantages in 2D/3D audio systems design. Implementation results are presented and discussed. As conclusions, challenges and future works are presented.

Nos últimos anos os avanços em áudio multicanal e sistemas envolventes despertaram um grande interesse pelas técnicas para a auralização de campos sonoros espaciais, capazes de recriar ambientações acústicas com grande realismo, envolvendo o ouvinte completamente. Um estudo sobre as tecnologias e soluções para áudio espacial mostrou que a construção de auralizadores envolve quatro blocos funcionais fundamentais. Mas, embora existam muitas técnicas e ferramentas disponíveis, não há uma arquitetura integradora para produção de áudio espacial que permita ao desenvolvedor selecionar as técnicas que deseja e montar uma solução com um grau de refinamento arbitrário que atenda a seus requisitos. Apresenta-se uma arquitetura modular aberta em quatro camadas para produção de áudio espacial em ambientes audiovisuais imersivos. Como estudo de caso, aborda-se a implementação de um auralizador Ambisonics para a CAVERNA Digital. Verifica-se que a auralização pode fornecer a correta perspectiva acústica do ambiente virtual necessária para uma percepção mais realista do espaço, e que a abordagem aberta para se implementar sistemas de áudio 2D/3D apresenta vantagens. Os resultados da implementação são apresentados e discutidos. Como conclusões, identificam-se desafios na implementação e os trabalhos futuros. / Recent advances in multichannel audio and surround systems have contributed to an increasing interest for spatial sound field auralization, capable of delivering acoustic ambience with great realism, and surrounding listener completely. A study on technologies and solutions for spatial sound has shown that building auralizators involve four major functional blocks, but, although there are many techniques and tools available, there is not an integration architecture for spatial audio production which permits developers to select their favorite techniques, and build a solution with an arbitrary refinement level, as wished. It is presented a four level modular open architecture for spatial audio production in immersive audiovisual environments. As a case study, an Ambisonics auralizator is implemented for the CAVERNA Digital CAVE. It is verified that auralization can deliver the correct acoustical perspective for the virtual environment necessary for a more realistic perception of space, and that an open approach presents advantages in 2D/3D audio systems design. Implementation results are presented and discussed. As conclusions, challenges and future works are presented.

Virtual Reality is one of the next big things in gaming, more and more games delivering an immersive VR-experience are popping up. Words such as immersion and presence has quickly become buzzwords that’s often used to describe a VR-game or experience. This interactive simulation of reality is literally turning people’s heads. The crowd pleaser, the ability to look around in 360-degrees, is however casting a shadow on the aural aspect. This study focused on this problem in relation to audio narrative. We examined which differences we could identify between a purely diegetic audio narrative and one utilizing a mix between diegetic and non-diegetic sound. How to grab the player’s attention and guide them to places in order for them to progress in the story. By spatializing audio using HRTF, we tested this dilemma through a game comparison with the help of soundscapes by R. Murray Schafer and auditory hierarchy by David Sonnenschein, as well as inspiration from Actor Network Theory. In our game comparison we found that while the synthesized sound, non-diegetic, ensured that the sound grabs the player’s attention, the risk of breaking the player’s immersion also increases.

Les technologies nomades (smartphones, tablettes, . . . ) étant actuellement très répandues,nous avons souhaité, dans le cadre de cette thèse, les utiliser comme vecteur pour proposer au grand public des outils de spatialisation sonore. La taille et le nombre de transducteurs utilisés pour la captation et la restitution sonore spatialisée sont à ce jour la limitation principale pour une utilisation nomade. Dans une première étape, la captation d’un opéra pour une restitution sur des tablettes tactiles nous a permis d’évaluer les technologies audio 3D disponibles aujourd’hui. Les résultats de cette évaluation ont révélé que l’utilisation des quatre capteurs du microphone Soundfield donne de bons résultats à condition d’effectuer un décodage binaural adapté pour une restitution sur casque. Selon une approche inspirée des méthodes de localisation de source et le concept de format « objet », un prototype de prise de son 3D léger et compact a été développé. Le dispositif microphonique proposé se compose de trois capsules microphoniques cardioïdes. A partir des signaux microphoniques, un algorithme de post-traitement spatial est capable, d’une part, de déterminer la direction des sources et, d’autre part, d’extraire un signal sonore représentatif de la scène spatiale. Ces deux informations permettent ainsi de caractérisercomplètement la scène sonore 3D en fournissant un encodage spatial offrant le double avantage d’une compression de l’information audio et d’une flexibilité pour le choix du système de reproduction. En effet, la scène sonore ainsi encodée peut être restituée en utilisant un décodage adapté sur n’importe quel type de dispositif.Plusieurs méthodes de localisation et différentes configurations microphoniques (géométrie et directivité) ont été étudiées.Dans une seconde étape, l’algorithme d’extraction de l’information spatiale a été modifié pour prendre en compte les caractéristiques réelles in situ des microphones.Des méthodes pour compléter la chaîne acoustique sont proposées permettant la restitution binaurale ainsi que sur tout autre dispositif de restitution. Elles proposent l’utilisation de capteurs de localisation présents sur les terminaux mobiles afin d’exploiter les capacités qu’ils offrent aujourd’hui. / Mobile technologies (such as smartphones and tablets) are now common devices of the consumer market. In this PhD we want to use those technologies as the way to introduce tools of sound spatialization into the mass market. Today the size and the number of traducers used to pick-up and to render a spatial sound scene are the main factors which limit the portability of those devices. As a first step, a listening test, based on a spatial audio recording of an opera, let us to evaluate the 3D audio technologies available today for headphone rendering. The results of this test show that, using the appropriate binaural decoding, it is possible to achieve a good binaural rendering using only the four sensors of the Soundfield microphone.Then, the steps of the development of a 3D sound pick-up system are described. Several configurations are evaluated and compared. The device, composed of 3 cardioid microphones, was developed following an approach inspired by the sound source localization and by the concept of the "object format encoding". Using the microphone signals and an adapted post-processing it is possible to determine the directions of the sources and to extract a sound signal which is representative of the sound scene. In this way, it is possible to completely describe the sound scene and to compress the audio information.This method offer the advantage of being cross platform compatible. In fact, the sound scene encoded with this method can be rendered over any reproduction system.A second method to extract the spatial information is proposed. It uses the real in situ characteristics of the microphone array to perform the sound scene analysis.Some propositions are made to complement the 3D audio chain allowing to render the result of the sound scene encoding over a binaural system or any king of speaker array using all capabilities of the mobile devices.

This thesis is related to the field of Sound Source Separation (SSS). It addresses the development and evaluation of these techniques for their application in the resynthesis of high-realism sound scenes by means of Wave Field Synthesis (WFS). Because the vast majority of audio recordings are preserved in twochannel stereo format, special up-converters are required to use advanced spatial audio reproduction formats, such as WFS. This is due to the fact that WFS needs the original source signals to be available, in order to accurately synthesize the acoustic field inside an extended listening area. Thus, an object-based mixing is required. Source separation problems in digital signal processing are those in which several signals have been mixed together and the objective is to find out what the original signals were. Therefore, SSS algorithms can be applied to existing two-channel mixtures to extract the different objects that compose the stereo scene. Unfortunately, most stereo mixtures are underdetermined, i.e., there are more sound sources than audio channels. This condition makes the SSS problem especially difficult and stronger assumptions have to be taken, often related to the sparsity of the sources under some signal transformation. This thesis is focused on the application of SSS techniques to the spatial sound reproduction field. As a result, its contributions can be categorized within these two areas. First, two underdetermined SSS methods are proposed to deal efficiently with the separation of stereo sound mixtures. These techniques are based on a multi-level thresholding segmentation approach, which enables to perform a fast and unsupervised separation of sound sources in the time-frequency domain. Although both techniques rely on the same clustering type, the features considered by each of them are related to different localization cues that enable to perform separation of either instantaneous or real mixtures.Additionally, two post-processing techniques aimed at improving the isolation of the separated sources are proposed. The performance achieved by several SSS methods in the resynthesis of WFS sound scenes is afterwards evaluated by means of listening tests, paying special attention to the change observed in the perceived spatial attributes. Although the estimated sources are distorted versions of the original ones, the masking effects involved in their spatial remixing make artifacts less perceptible, which improves the overall assessed quality. Finally, some novel developments related to the application of time-frequency processing to source localization and enhanced sound reproduction are presented. / Cobos Serrano, M. (2009). Application of sound source separation methods to advanced spatial audio systems [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/8969 / Palancia

This thesis project aims for the development of a demonstration platform on mobile devices for testing and demonstrating algorithms for 2D and 3D spatial sound reproduction. The demo system consists of four omnidirectional microphones in a square planar array, an Octo sound card (from Audio Injector), a Raspberry Pi 3B+ (R-Pi) single-board computer and an inertial measurement unit (IMU) located in the center of the array. The microphone array captures sound, which is then digitized, and in turn, transferred to the R-Pi. On the R-Pi, the digitized sound signal is rendered through the directional audio coding (DirAC) algorithm to maintain the spatial properties of the sound. Finally, the digital signal and spatial properties are rendered through Dirac VR to maintain a spatial stereo signal of the recorded environment. The directional audio coding algorithm was initially implemented in Matlab and then ported to C++ since the R-Pi does not support Matlab natively. The ported algorithm was verified on a four-channel in and six-channel out system, processing 400 000 samples at 44 100 kHz. The results show that the C++ DirAC implementation maintained a maximum error of 4.43e-05 or -87 dB compares to the original Matlab implementation. For future research on spatial audio reproduction, a four-microphone smartphone mock-up was constructed based on the same hardware used in the demo system. A software interface was also implemented for transferring the microphone recordings and the orientation of the mock-up to Matlab.

A key aspect of immersive audio is realistic acoustics. To get plausible acoustics for an environment the impulse response can be generated using acoustic simulations and should ideally be updated in real-time for interactive applications. In this thesis the listening experience of sound generated with an interactive sound propagation engine was explored and compared to spatial sound produced with a static impulse response. The aim was to evaluate the sound experience for applications outside of virtual reality, with computational cost in consideration. This was done by conducting a user study where the participants got to interact and compare the two sound methods in different environments. The study was performed using a custom developed application integrated with a pre-existing sound propagation engine. The results from the user study showed no obvious perceptive difference between the two sound rendering methods that could justify the extra computations. Overall there was even a slight preference for the stereo method that used a static impulse response. However, there were qualities to both sound rendering methods that were preferred depending on the environment. Another thing that was investigated in the work of this thesis was how the varying accuracy of localization of sound in different directions can be used in acoustic ray tracing algorithms. An alternative sampling method was developed that uses a biased distribution based on spatial resolution of human hearing instead of traditional uniform sampling. The computation time of the random sampling phase increases, but could potentially reduce the number of ray samples needed. / <p>Examensarbetet är utfört vid Institutionen för teknik och naturvetenskap (ITN) vid Tekniska fakulteten, Linköpings universitet</p>