Transforming human movement into live sound can be used as a method to enhance motor skill learning via the provision of augmented perceptual feedback. A small but growing number of studies hint at the substantial efficacy of this approach, termed 'movement sonification'. However there has been sparse discussion in Psychology about how movement should be mapped onto sound to best facilitate learning. The current thesis draws on contemporary research conducted in Psychology and theoretical debates in other disciplines more directly concerned with sonic interaction - including Auditory Display and Electronic Music-Making - to propose an embodied account of sonification as feedback. The empirical portion of the thesis both informs and tests some of the assumptions of this approach with the use of a custom bimanual coordination paradigm. Four motor skill learning studies were conducted with the use of optical motion-capture. Findings support the general assumption that effective mappings aid learning by making task-intrinsic perceptual information more readily available and meaningful, and that the relationship between task demands and sonic information structure (or, between action and perception) should be complementary. Both the theoretical and empirical treatments of sonification for skill learning in this thesis suggest the value of an approach which addresses learner experience of sonified interaction while grounding discussion in the links between perception and action.

A sonification is a nonverbal speech act and might sound like “wa-UM-eii”, or “wooosh!” The purpose of this study was to investigate a choreographer’s use of sonification in dance instructions, to see if there are different types of sonifications and if the use of these might differ with a change in context. Video material capturing the rehearsals of a noted dance company was analyzed using a cognitive ethnography-based approach. Nine different types of sonifications were identified and described according to purpose, and a context-based analysis showed that certain kinds of sonifications occurred more frequently in some contexts than others. The results suggest that sonification used in dance instruction can serve multiple different purposes – the three main purposes described here are these: to communicate the quality of a movement, to facilitate communication between choreographer and dancers, and to coordinate the dancers as a group.

Auditory graphsdisplays that represent graphical, quantitative information with soundhave the potential to make graphical representations of data more accessible to blind students and researchers as well as sighted people. No research to date, however, has systematically addressed the attributes of data that contribute to the complexity (the ease or difficulty of comprehension) of auditory graphs. A pair of studies examined the role of both data density (i.e., the number of discrete data points presented per second) and the number of trend reversals for both point estimation and trend identification tasks with auditory graphs. For the point estimation task, results showed main effects of both variables, with a larger effect attributable to performance decrements for graphs with more trend reversals. For the trend identification task, a large main effect was again observed for trend reversals, but an interaction suggested that the effect of the number of trend reversals was different across lower data densities (i.e., as density increased from 1 to 2 data points per second). Results are discussed in terms of data sonification applications and rhythmic theories of auditory pattern perception.

nÃo hà / O uso de diferentes mÃtodos de sonificaÃÃo de dados meteorolÃgicos e geofÃsicos à recorrente tanto em aplicaÃÃes cientÃficas quanto no contexto da criaÃÃo artÃstica. PropÃe-se, atravÃs de diferentes abordagens ao longo do processo de pesquisa, a criaÃÃo de uma sÃrie de sintetizadores e dispositivos sonoros sensÃveis à variaÃÃes de intensidade do vento, da luminosidade e da temperatura ambiente. Com o intuito de explorar as potencialidades do uso da sonificaÃÃo de dados de forÃas da natureza como mÃtodo de composiÃÃo sonora e musical, utilizam-se diversos sensores para controlar parÃmetros distintos do processo de sÃntese sonora digital. AtravÃs da experimentaÃÃo de circuitos e cÃdigos de programaÃÃo em Arduino, busca-se analisar os fatores que influenciam diferentes aspectos do processo de composiÃÃo junto aos fenÃmenos meteorolÃgicos. à apresentado o processo de desenvolvimento de trÃs sintetizadores que sÃo utilizados em diferentes ocasiÃes, alÃm da construÃÃo de uma mÃquina sonificadora e a produÃÃo de registros diversos do processo de elaboraÃÃo dos protÃtipos. Com base nas experimentaÃÃes realizadas, identificam-se aspectos pertinentes à atuaÃÃo de elementos da natureza como forÃas criativas durante o processo de composiÃÃo sonora e musical. / The use of different methods of sonification of meteorological and geophysical data is recurrent both in scientific applications and in the context of artistic creation. It is proposed, through different approaches throughout the research process, the creation of a series of synthesizers and sound devices sensitive to variations of wind intensity, luminosity and ambient temperature. In order to explore the potentialities of using sonification of forces of nature as a method of sound and musical composition, several sensors are used to control different parameters of the digital sound synthesis process. Through the experimentation of circuits and programming codes in Arduino, it is sought to analyze the factors that influence different aspects of the composition process with the meteorological elements. The process of development of three synthesizers that are used in different occasions is presented, along with the construction of a sonifying machine and the production of diverse material about the process of elaboration of the prototypes. Based on the experiments carried out, aspects pertinent to the performance of elements of nature as creative forces during the process of sound and musical composition are identified.

Human auditory perception is suited to receiving and interpreting information from the environment but this knowledge has not been used extensively in designing computer-based information exploration tools. It is not known which aspects of sound are useful for accurately conveying information in an auditory display. An auditory display was created using PD, a graphical programming language used primarily to manipulate digital sound. The interface for the auditory display was a blank window. When the cursor is moved around in this window, the sound generated would changed based on the underlying data value at any given point. An experiment was conducted to determine which attribute of sound most accurately represents data values in an auditory display. The four attributes of sound tested were frequency-sine waveform, frequency-sawtooth waveform, loudness and tempo. 24 subjects were given the task of finding the highest data point using sound alone using each of the four sound treatments. Three dependent variables were measured: distance accuracy, numeric accuracy, and time on task. Repeated measures ANOVA procedures conducted on these variables did not rise to the level of statistical significance (α=.05). None of the sound treatments was more accurate than the other as representing the underlying data values. 52% of the trials were accurate within 50 pixels of the highest data point (target). An interesting finding was the tendency for the frequency-sin waveform to be used in the least accurate trial attempts (38%). Loudness, on the other hand, accounted for very few (12.5%) of the least accurate trial attempts. In completing the experimental task, four different search techniques were employed by the subjects: perimeter, parallel sweep, sector, and quadrant. The perimeter technique was the most commonly used.

Shoplifting is a serious issue that causes loss to retail owners. However, the loss can be avoided by adding preventative measures to deter shoplifters from committing the crime. Surveillance [2] and a store's design and layout [3] are considered significant factors that could deter them. Considering that background music is a part of a store's design, this research combined surveillance and a store's design to discourage shoplifters by creating changes to the background music that could make the shoplifters feel observed. Ultimately, this research asked these questions: 1) If a change in background music could make the visitor feel observed, what are the possible design strategies? and 2) which design is preferable? In this research, eight participants watched a first-person view of a shopper going around the shop and lifting a clothing article. The participants were asked to create a sound alert that matched the lifting action, mixed with three different background music tracks. Afterwards, they were asked to hand in the documentation of their design. Apart from that, they were asked to fill in a quantitative survey evaluating all the designs (including other participants' designs) and describe the design with three keywords. This research discovered that there were three main approaches in designing the sound alert, and there were certain preferences of a design approach dependent on the type of background music. In other words, there is a correlation between the type of background music and the preferred design approach. This work also discussed its limitations and presented opportunities for future research. / Snatteri är ett allvarligt problem som orsakar förluster för butiksägare. Förlusterna kan dock undvikas genom förebyggande åtgärder för att avskräcka butikstjuvarna från att begå brottet. Övervakning [2] samt butikens design och planlösning [3] anses vara viktiga faktorer som kan avskräcka dem. Med tanke på att bakgrundsmusik är en del av en butikens design kombinerar denna forskning övervakning och butikens design för att avskräcka snattare genom att skapa förändringar i bakgrundsmusiken som kan få snattarna att känna sig observerade. Avslutningsvis ställde den här forskningen dessa frågor: 1) Om en förändring av bakgrundsmusiken kan få besökaren att känna sig observerad, vilka är då de möjliga designstrategierna? och 2) Vilken design är att föredra? I den här forskningen tittade åtta deltagare på en shoppare från hen som går runt i butiken och lyfter ett klädesplagg. Deltagarna ombads att skapa ett ljudlarm som matchade lyfthandlingen, blandat med tre olika bakgrundsmusikspår. Efteråt ombads de att lämna in dokumentationen av sin design. Dessutom ombads de att fylla i en kvantitativ enkät där de utvärderade alla konstruktioner (inklusive andra deltagares konstruktioner) och beskrev konstruktionen med tre nyckelord. Forskningen visade att det fanns tre huvudsakliga tillvägagångssätt vid utformningen av ljudvarningen, och att det fanns vissa preferenser för ett tillvägagångssätt beroende på typen av bakgrundsmusik. Med andra ord finns det en korrelation mellan typen av bakgrundsmusik och den föredragna designmetoden. I detta arbete diskuterades också dess begränsningar och presenterades möjligheter för framtida forskning.

Sonification is the systematic transformation of data to sound. Sonification is a means to communicate information, to support navigation, or to explore data. In a multi-disciplinary research project we develop a psychoacoustically-motivated sonification that supports surgeons’ orientation in image-guided interventions. One drawback of image-guidance is that neither the location of monitors nor the displayed view on the patient’s anatomy coincides with the actual viewpoint of the surgeon. Surgeons need to mentally scale, rotate, and translate the displayed graphics. These operations are cognitively demanding. Sonification can reduce cognitive load and relieve the visual channel to improve the ergonomic situation in the operating room by communicating the location of the target, relative to the tool tip, like the center of a tumor relative to the ablation needle. By means of psychoacoustic sonification we ensure that the sounds are readily interpretable in terms of orthogonal and linear dimensions with a high resolution.

This thesis presents an investigation of how to design engaging audio games. This was done by creating two prototypes. The first prototype was a rhythm based fighting game. As it was being developed it became clear that it did not differentiate itself enough from a large amount of already existing audio games. The second prototype was a first person shooter/adventure game utilizing 3D audio. The second prototype was evaluated by observing users playing the prototype and having them answer two questionnaires followed by an interview. Using spatial audio opens up a lot of possibilities. First person navigation using spatial audio is clearly possible, although how to implement it with a high degree of playability is in need of further study. When designing a first person audio game it is important to keep in mind that navigating, aiming etc. is more difficult than in video games. A slower pace seems suitable to this type of game. A lot of time and effort has to be put into the details when designing the gameplay of an audio game. Seemingly inconsequential things can be detrimental to the player’s enjoyment. Focus should be on perfecting the fundamental actions and challenges. / <p>Examensarbetet är utfört vid Institutionen för teknik och naturvetenskap (ITN) vid Tekniska fakulteten, Linköpings universitet</p>

Over the last five years I have been investigating the phenomenon of movement sonification and different types of interaction ongoing within the experience. I have expanded my knowledge and skills in the fields of music, musical performance, dance, music theory, music technology, acoustics, physics, as well as cognitive and computer sciences.  This paper introduces the reader to an extraordinarily broad subject of movement sonification, namely how to create, participate in and understand the phenomenon, with special emphasis on the human perception and proprioception system. Particular attention is also paid to technological issues, presented and described using the example of IMU technology.  The thesis presents also my point of view on direct and indirect interaction through examples of my previous compositions, as well as works created by other artists. The two last chapters are devoted to practical aspects – experiences and observations made from different perspectives – a composer’s, creator’s, performer’s and spectator’s. Based on my latest interactive work Hypercycle I discuss my way of creating an interactive situation, the process of using all possible elements to make the work engaging, yet not overwhelming or overstimulating. / <p>The documentation also includes the following recordings: Hypercycle (2021) – documentation – visual instructions.mp4; Hypercycle (2021) – documentation – binaural.mp4; Hypercycle (2021) - exhibition (frag.).mov</p>

Visualisering har vid flertaliga tillfällen visats användbart och effektivt inom undervisning, särskilt för att ge studenter en intuitiv förståelse för abstrakta ämnen och problem. Studiens syfte är att undersöka hur visualisering kan appliceras inom signalhantering och filter. Ett visualiseringsverktyg har utvecklats i enlighet med teorier om interaktivitet och gränssnitt för utbildning. Sedan utfördes en användarstudie med sammanlagt 14 deltagare, varav hälften utgjorde en kontrollgrupp. Visualiseringsverktyg effekt på deltagarnas förståelse för signalfilter och förmåga att absorbera ny information mättes. Deltagarnas kunskapsökning mättes genom att deras resultat på ett avslutande kunskapsprov jämfördes med deras prestation på en inledande kunskapsdiagnos. Inom studiens ramar kunde det inte påvisas att deltagarnas kunskap förbättrades tack vare visualisering. De viktigaste elementen i ett visualiseringsverktygs gränssnitt borde framhävas ännu mer än i verktyget som användes i denna studie. Däremot gillade en majoritet av användarna verktyget och flera tyckte att de fick nya perspektiv på eller nytt intresse för ämnet som studien avhandlade. Multimodala verktyg som erbjuder fri och naturlig interaktion är ett bra sätt att öka studenters intresse för komplicerade och abstrakta ämnen. / Visualization has repeatedly been proven to be an effective tool to let students develop an intuitive understanding of abstract problems. This study aims to find out how visualization can enhance the teaching of signal processing. For this purpose, a visualization tool was developed according to theories about interactive teaching objects. Then, a user study was performed, which 14 people participated in. One half made up a control group. To measure the effect the tool had on performance, the participants completed a test before and after using the tool, and their results were compared. Within the scope of this study, no benefit from visualization to performance could be found. However, a majority of the participants enjoyed the visualization tool, and liked the way it gave them new ways of looking at signal processing. The study concludes that visualization is a good method to make abstract and difficult topics more approachable for students.