This thesis presents a multimodal sonification system that combines video with sound synthesis generated from motion capture data. Such a system allows for a fast and efficient exploration of musicians' ancillary gestural data, for which sonification complements conventional videos by stressing certain details which could escape one's attention if not displayed using an appropriate representation. The main objective of this project is to provide a research tool designed for people that are not necessarily familiar with signal processing or computer sciences. This tool is capable of easily generating meaningful sonifications thanks to dedicated mapping strategies. On the one hand, the dimensionality reduction of data obtained from motion capture systems such as the Vicon is fundamental as it may exceed 350 signals describing gestures. For that reason, a Principal Component Analysis is used to objectively reduce the number of signals to a subset that conveys the most significant gesture information in terms of signal variance. On the other hand, movement data presents high variability depending on the subjects: additional control parameters for sound synthesis are offered to restrain the sonification to the significant gestures, easily perceivable visually in terms of speed and path distance. Then, signal conditioning techniques are proposed to adapt the control signals to sound synthesis parameter requirements or to allow for emphasizing certain gesture characteristics that one finds important. All those data treatments are performed in realtime within one unique environment, minimizing data manipulation and facilitating efficient sonification designs. Realtime process also allows for an instantaneous system reset to parameter changes and process selection so that the user can easily and interactively manipulate data, design and adjust sonifications strategies.

No description available.

Thesis (M. Phil.)--University of Stellenbosch, 2005. / ENGLISH ABSTRACT: The field of gesture based research and the interaction between human and computer with the focus falling on musical applications, is well established internationally. However, in South Africa, research in this field appears dormant. The reasons for this state of affairs are complex and can be argued from different angles covering socio-economical, philosophical and educational perspectives. This document describes the design, creation and implementation of an operational gesture interface environment which holds the potential to be expanded in the future. The implementation draws on cost-efficient hard- and software in the design of elementary to more advanced musical and even non-musical virtual environments (VEs) harbouring potential for further research and performance. Hard- and software available at Stellenbosch University’s Konservatorium were used together with selected free downloadable software from the internet in creating VEs, which to a degree simulate other techniques of sound manipulation. The choice of software was guided by the availability of support and prominence in terms of usage. Software basically had to incorporate hand movement tracking and the mapping of data to manipulate several parameters. Three independent systems, each representing a different VE, were studied, experimented with and programmed in order to validate the thesis. The first system manipulates a complete electronic musical instrument. The second system incorporates the simulation of a real-life musical performance and the third system focuses on manipulating specific sequencing software by a basic alternative computer mouse implementation. The outcome of this thesis provides an environment within which several programming techniques are treated and combined to form a template for teaching this field, and future development and research. These techniques incorporate the manipulation of digital audio, deal with a digital communication protocol, basic computer graphics and other necessary programming algorithms. In addition, the thesis strives to provide an outline for the understanding, design and implementation of a VE installation. The three systems will be installed for operation during a presentation of this thesis. Together with the three operative systems, this document strives to act as an initial platform from which exciting futuristic research and activity can be launched. / AFRIKAANSE OPSOMMING: Die gebied van beweging gebaseerde navorsing en die interaksie tussen mens en rekenaar waar die fokus op musikale toepassings val, is internasionaal stewig gevestig. In Suid-Afrika egter, blyk navorsing op hierdie gebied sluimerend te wees. Die redes vir hierdie stand van sake is kompleks en kan vanuit verskillende hoeke wat sosio-ekonomiese, filosofiese en opvoedkundige perspektiewe insluit, beredeneer word. Hierdie dokument beskryf die ontwerp, skep en implementering van ‘n operasionele bewegings koppelvlak omgewing met potensiaal tot uitbreiding. Die implementering baseer op kosteeffektiewe hardeware en programmatuur in die ontwerp van eenvoudige tot gevorderde virtuele omgewings (VOs) vir musiek, en selfs nie-musikale dissiplines met die potensiaal tot verdere navorsing en implementering binne die musikale uitvoeringspraktyk. Hardeware en programmatuur beskikbaar aan die Konservatorium is gebruik tesame met ‘n seleksie van gratis programmatuur op die internet beskikbaar om VOs wat ander klankmanipulerings tegnieke simuleer, te skep. Die keuse van programmatuur is gelei deur die beskikbaarheid van ondersteuning en gewildheid en inkorpeer die volg van hand beweging asook die verspreiding van data om verskeie parameters te manipuleer. Drie onafhanklike sisteme wat elk ‘n ander VO voorstel, is bestudeer, mee ge-ekperimenteer en geprogrammeer om die tesis te valideer. Die eerste sisteem manipuleer ‘n volledige elektroniese musiekinstrument. Die tweede sisteem inkorporeer die simulasie van ‘n werklike musikale uitvoering en die derde sisteem fokus op die manipulasie van spesifieke sequencing programmatuur sonder die hulp van ‘n muis. Die uitkoms van hierdie tesis verskaf ‘n omgewing waarbinne heelparty programmerings tegnieke bespreek en gekombineer word in ‘n aanpasbare templaat vir die onderrig van hierdie veld asook toekomstige ontwikkeling en navorsing. Hierdie tegnieke inkorporeer die manipulasie van digitale klank, die omgang met ‘n digitale kommunikasie protokol, basiese rekenaargrafika en verdere noodsaaklike programmerings algoritmes. Verder streef die tesis daarna om ‘n raamwerk vir die begrip, ontwerp en implementering van ‘n VO daar te stel. Die drie sisteme wat in hierdie tesis bespreek word, sal operasioneel geïnstalleer word gedurende ‘n demonstrasie daarvan. Saam met die drie werkende sisteme streef hierdie dokument daarna om te dien as platform waarvan af opwindende futuristiese navorsing en aktiwiteite geïniseer kan word.