Visualisering har vid flertaliga tillfällen visats användbart och effektivt inom undervisning, särskilt för att ge studenter en intuitiv förståelse för abstrakta ämnen och problem. Studiens syfte är att undersöka hur visualisering kan appliceras inom signalhantering och filter. Ett visualiseringsverktyg har utvecklats i enlighet med teorier om interaktivitet och gränssnitt för utbildning. Sedan utfördes en användarstudie med sammanlagt 14 deltagare, varav hälften utgjorde en kontrollgrupp. Visualiseringsverktyg effekt på deltagarnas förståelse för signalfilter och förmåga att absorbera ny information mättes. Deltagarnas kunskapsökning mättes genom att deras resultat på ett avslutande kunskapsprov jämfördes med deras prestation på en inledande kunskapsdiagnos. Inom studiens ramar kunde det inte påvisas att deltagarnas kunskap förbättrades tack vare visualisering. De viktigaste elementen i ett visualiseringsverktygs gränssnitt borde framhävas ännu mer än i verktyget som användes i denna studie. Däremot gillade en majoritet av användarna verktyget och flera tyckte att de fick nya perspektiv på eller nytt intresse för ämnet som studien avhandlade. Multimodala verktyg som erbjuder fri och naturlig interaktion är ett bra sätt att öka studenters intresse för komplicerade och abstrakta ämnen. / Visualization has repeatedly been proven to be an effective tool to let students develop an intuitive understanding of abstract problems. This study aims to find out how visualization can enhance the teaching of signal processing. For this purpose, a visualization tool was developed according to theories about interactive teaching objects. Then, a user study was performed, which 14 people participated in. One half made up a control group. To measure the effect the tool had on performance, the participants completed a test before and after using the tool, and their results were compared. Within the scope of this study, no benefit from visualization to performance could be found. However, a majority of the participants enjoyed the visualization tool, and liked the way it gave them new ways of looking at signal processing. The study concludes that visualization is a good method to make abstract and difficult topics more approachable for students.

Immersive analytics is an emerging field of data exploration and analysis in immersive environments. It is an active research area that explores human-centric approaches to data exploration and analysis based on the spatial arrangement and visualization of data elements in immersive 3D environments. The availability of immersive extended reality systems has increased tremendously recently, but it is still not as widely used as conventional 2D displays. In this dissertation, we described an immersive analysis system for spatiotemporal data and performed several user studies to measure the user performance in the developed system, and laid out design guidelines for an immersive analytics environment. In our first study, we compared the performance of users based on specific visual analytics tasks in an immersive environment and on a conventional 2D display. The approach was realized based on the coordinated multiple-views paradigm. We also designed an embodied interaction for the exploration of spatial time series data. The findings from the first user study showed that the developed system is more efficient in a real immersive environment than using it on a conventional 2D display. One of the important challenges we realized while designing an immersive analytics environment was to find the optimal placement and identification of various visual elements. In our second study, we explored the iterative design of the placement of visual elements and interaction with them based on frames of reference. Our iterative designs explored the impact of the visualization scale for three frames of reference and used the collected user feedback to compare the advantages and limitations of these three frames of reference. In our third study, we described an experiment that quantitatively and qualitatively investigated the use of sonification, i.e., conveying information through nonspeech audio, in an immersive environment that utilized empirical datasets obtained from a multi-dimensional geophysical system. We discovered that using event-based sonification in addition to the visual channel was extremely effective in identifying patterns and relationships in large, complex datasets. Our findings also imply that the inclusion of audio in an immersive analytics system may increase users’ level of confidence when performing analytics tasks like pattern recognition. We outlined the sound design principles for an immersive analytics environment using real-world geospace science datasets and assessed the benefits and drawbacks of using sonification in an immersive analytics setting. / Doctor of Philosophy / When it comes to exploring data, visualization is the norm. We make line charts, scatter plots, bar graphs, or heat maps to look for patterns in data using traditional desktop-based approaches. However, biologically humans are optimized to observe the world in three dimensions. This research is motivated by the idea that representing data in immersive 3D environments can provide a new perspective that may lead to the discovery of previously undetected data patterns. Experiencing the data in three dimensions, engaging multiple senses like sound and sight, and leveraging human embodiment, interaction capabilities, and sense of presence may lead to a unique understanding of the data that is not feasible using traditional visual analytics. In this research, we first compared the data analysis process in a mixed reality system, where real and virtual worlds co-exist, versus doing the same analytical tasks in a desktop-based environment. In our second study, we studied where different charts and data visualizations should be placed based on the scale of the environment, such as table-top versus room-sized. We studied the strengths and limitations of different scales based on the visual and interaction design of the developed system. In our third study, we used a real-world space science dataset to test the liabilities and advantages of using the immersive approach. We also used audio and explored what kinds of audio work for which analytical tasks and laid out design guidelines based on audio. Through this research, we studied how to do data analytics in emerging mixed reality environments and presented results and design guidelines for future developers, designers, and researchers in this field.

This dissertation explains two sonification platforms designed for image-sound association study and art, and the study results. A platform for user study was developed first and an artistic audiovisual platform was derived based on it. First, the five image-sound association studies were conducted to see whether people can successfully associate sounds and fundamental shapes (i.e., a circle, a triangle, a square, lines, curves, and other custom shapes) and the correct answer rate was high. Then, the same sonification platform was transformed by adding colors to the audiovisual platform for artistic/musical expression. A line-by-line sonification method and an object-oriented method were newly developed to sonify the background and the shapes separately. To enhance user experience, the sound of each shape was spatialized in a multi-layer speaker environment or a virtual listening environment. / Doctor of Philosophy / This dissertation introduces two sonification platforms designed for image-sound association study and art, and the study results. A platform for user study was developed first and an artistic audiovisual platform was derived based on it. First, the five image-sound association studies were conducted to see whether people can successfully associate sounds and fundamental shapes (i.e., a circle, a triangle, a square, lines, curves, and other custom shapes) and the correct answer rate was high. Then, the same sonification platform was transformed by adding colors to the audiovisual platform for artistic/musical expression. A line-by-line sonification method and an object-oriented method were newly developed to sonify the background and the shapes separately. To enhance user experience, the sound of each shape was spatialized in a multi-layer speaker environment or a virtual listening environment.

In recent years, advances in deep learning have allowed various forms of electrographic signals, such as electroencephalography (EEG) and electromyography (EMG), to be used as a viable form of input in artificial intelligence applications, particularly for applications in the medical field. One such topic that EMG inputs have been used is in silent speech interfaces, or devices capable of processing speech without an audio-based input. The goal of this thesis is to explore a novel method of training a machine learning model to be used for silent speech interface development: using transfer learning to leverage a pre-trained speech recognition model for classifying EMG-based silent speech inputs. To accomplish this, we pass labeled EMG data through a custom transformation process, turning the data into musical notes that represent changes in an EMG sensor as silent speech data is captured. This transformed data was used as input into a pre-trained speech recognition model, and the model's classification layers were retrained to better fit the incoming data. The custom transformation process and model demonstrated progress towards effective classification with a small, closed-vocabulary dataset but showed no signs of effective training with a larger, open-vocabulary dataset. The effectiveness on the small closed-vocabulary dataset demonstrate that training a model to recognize EMG data using transfer learning on a pre-trained speech to text model is a viable approach.

In the modern computing landscape, the challenge of tuning software systems is exacerbated by the necessity to accommodate multiple divergent execution environments and stakeholders. Achieving optimal performance requires a different configuration for every combination of hardware setups and business requirements. In addition, the state of the art in system tuning can involve complex statistical models and tools which require deep expertise not commonly possessed by the average software engineer. As an alternative approach to performance tuning, this thesis puts forward the use of sonification-conveying information via non-speech audio-to aid software engineers in tuning complex systems. In particular, this thesis designs, develops, and evaluates a tuning system that interactively (i.e., in response to user actions) sonifies the performance metrics of a computer system. This thesis demonstrates that interactive sonification can effectively guide software engineers through performance tuning of a computer system. To that end, a scientific survey determined which sound characteristics (e.g., loudness, panning, pitch, tempo, etc.) are best suited to express information to the engineer. These characteristics were used to create a proof-of-concept tuning system that was applied to tune the parameters of a real world enterprise application server. Equipped with the tuning system, engineers-not experts in enterprise computing nor performance tuning-were able to tune the server, so that its resulting performance surpasses that exhibited under the standard configuration. The results indicate that sound-based tuning approaches can provide valuable solutions to the challenges of configuring complex computer systems. / Master of Science

Afin de réduire les consommations de carburant et les émissions de CO2, des technologies comme la désactivation de cylindres ont été développées. Deux stratégies ont été à l’étude chez le Groupe PSA, il s’agit de désactivations appelées fixe ou tournante en fonction du nombre de cylindres désactivés au cours d’un cycle moteur. Des modifications importantes du bruit et des vibrations transmises dans l’habitacle de la voiture en sont les conséquences ce qui modifie nettement la perception du confort par le conducteur de la voiture. Ce travail de recherche s’intéresse aux modifications de confort perçu par le conducteur dues à ces différentes stratégies appliquées au moteur ainsi qu’aux solutions potentielles qui permettraient d’améliorer ce confort. Parmi ces solutions se trouve le principe de sonification en temps réel du bruit moteur. Pour répondre à ces questions, cinq tests perceptifs sont réalisés. Les deux premiers consistent en l’évaluation du confort global dans différentes configurations ainsi qu’à la validation de l’utilisation d’un simulateur vibro-acoustique. Ces expériences ont montré qu’une des stratégies de désactivation était jugée significativement plus inconfortable que les autres. Par la suite, une seule des stratégies sera retenue. La troisième expérience permet de déterminer un seuil d’acceptabilité entre les sollicitations du moteur avec désactivation et celles du moteur habituel. Ceci afin de fixer une cible pour laquelle l’environnement vibro-acoustique serait acceptable. Dans les deux dernières expériences, deux solutions permettant d’atteindre cette cible sont simulées. La première consiste à modifier la plage de régime moteur ou à lieu la désactivation, la deuxième consiste à assouplir les suspensions horizontales afin de limiter les résonances vibratoires à faible régime. / The technology of cylinder of deactivation has been developed in order to reduce gas consumption and CO2 emissions. Two strategies were studied at groupe PSA called fixed and rotating deactivations depending on the number of cylinders deactivated per engine cycle. This implies non neglecting modifications of sound and vibrations transmitted to the car cabin depredating the global comfort of the car. This research work focused on how driver’s comfort was altered by these engine configurations and how it would be possible to improve this comfort. Among the solutions possible, appears the principle of real-time sonification of the engine noise. To answer these questions, five perceptual experiments have been conducted. First, the aim was to evaluate global comfort with different engine configurations and validate the use of a vibro-acoustic simulator. This showed that one deactivation strategy was significantly reducing the comfort evaluation. Then, we focused on the second strategy which was also considered as not comfortable. On the third experiment, we were interested in finding an acceptable threshold between the vibrations and sounds with the deactivation and with the usual engine configuration. This led to a target of signal to reach in order to provide acceptable situations in terms of sound and vibrations. In the last two experiments, we were interested in the simulation of two solutions about the deactivation settings that would reduce the annoyance: the modification of the engine speed range in which the deactivation occurs and the softening of horizontal suspensions part in order to reduce vibrations resonances at low engine speed.

Cette thèse s'est intéressée aux relations entre les mouvements biologiques et la perception sonore en considérant le cas spécifique des mouvements graphiques et des sons de frottement qu'ils génèrent. L'originalité de ces travaux réside dans l'utilisation d'un modèle de synthèse sonore basé sur un principe perceptif issu de l'approche écologique de la perception et contrôlé par des modèles de gestes. Des stimuli sonores dont le timbre n'est modulé que par des variations de vitesse produites par un geste ont ainsi pu être générés permettant de se focaliser sur l'influence perceptive de cet invariant transformationel. Une première étude a ainsi montré que l'on reconnait la cinématique des mouvements biologiques (la loi en puissance 1/3), et que l'on peut discriminer des formes géométriques simples juste à partir des sons de frottement produits. Une seconde étude a montré l'existence de prototypes dynamiques sonores caractérisant les trajectoires elliptiques, mettant ainsi en évidence que les prototypes géométriques peuvent émerger d'un couplage sensorimoteur. Enfin, une dernière étude a montré qu'une cinématique évoquée par un sonore influence significativement la cinématique et la géométrie d'un geste dans une tâche de reproduction graphique du mouvement d'un point lumineux. Ce résultat révèle l'importance de la modalité auditive dans l'intégration multisensorielle des mouvements continus dans une situation jamais explorée. Ces résultats ont permis le contrôle de modèles de synthèse par des descriptions gestuelles et la création d'outils de sonification pour l'apprentissage de gestes et la réhabilitation d'une pathologie motrice, la dysgraphie. / This thesis focused on the relations between biological movements and auditory perception in considering the specific case of graphical movements and the friction sounds they produced. The originality of this work lies in the use of sound synthesis processes that are based on a perceptual paradigm and that can be controlled by gesture models. The present synthesis model made it possible to generate acoustic stimuli which timbre was directly modulated by the velocity variations induced by a graphic gesture in order to exclusively focus on the perceptual influence of this transformational invariant. A first study showed that we can recognize the biological motion kinematics (the 1/3 power law) and discriminate simple geometric shapes simply by listening to the timbre variations of friction sounds that solely evoke velocity variations. A second study revealed the existence of dynamic prototypes characterized by sounds corresponding to the most representative elliptic trajectory, thus revealing that prototypical shapes may emerged from sensorimotor coupling. A final study showed that the kinematics evoked by friction sounds may significantly affect the dynamic and geometric dimension in the visuo-motor coupling. This shed critical light on the relevance of auditory perception in the multisensory integration of continuous motion in a situation never explored. All of these theoretical results enabled the gestural control of sound synthesis models from a gestural description and the creation of sonification tools for gesture learning and rehabilitation of a graphomotor disease, dysgraphia.

Computer usage today is predominantly based on graphical interaction, where the visual presentation of information is essential both for input (hand-eye coordination when using a computer mouse), and output (seeing the information on a computer screen). This can create difficulties for blind computer users, both at an individual level when interacting with a computer, and also when collaborating with other computer users. The work presented in this thesis has investigated interaction for blind computer users in three stages. First investigating access to information by making studies on an interactive audio-only game, drawing conclusions about auditory direct manipulation and auditory interface design. Second studying collaboration between blind and sighted computer users in two different contexts, leading to questioning of commonly expressed design principles regarding access to collaboration. Finally studying accessibility in a working environment, finding out how technology, the assistive device used by the blind person, communication with others and professional knowledge interplayed to create an accessible work environment. Based on these empirical studies, the main conclusion from this work is a proposal of a research perspective, Assistive interfaces as cooperative interfaces. Here, the context where the interface is going to be used is in focus, and cooperative and social dimensions of interaction are acknowledged and highlighted. The design and analysis of assistive devices should be highly sensitive to the socio-interactional environment, and not just focusing on the single individual using an assistive device. / QC 20100921

In this project, I have developed and built a new prototype of the ColoroPhone visual aid. This new prototype has sensors to measure the distance to objects in front of the wearer, which then communicates with a mobile application that generates sounds depending on the distance to the object. The user will hear this sound through bone-conducting headphones This addition to the ColoroPhone is intended to further enhance the assistance the device can offer. The current prototype is a functioning version and will be used in further research done by a group of psychologists, which will study how such devices affect users both on a biological and social level. They will also evaluate how to approach user training. The prototype in its current state have several segments that have considerable room for improvement in further development of the platform. Some of these improvements can be in form of better Bluetooth implementation, better or more sensors, or improvements in the current set of sensors regarding noise handling and data processing. This type of device can have large impact on the daily life for a large and growing amount of people across the world, as independence is a large factor in a persons feel of self-worth and long term psychological health.

Orientadora: Profª. Drª. Márcia Aparecida da Silva Spinacé / Dissertação (mestrado) - Universidade Federal do ABC. Programa de Pós-Graduação em Ciência e Tecnologia/Química, 2015. / Na atualidade, tem aumentado o debate e interesse por processos de produção limpos e economicamente viáveis de celulose nanoestruturada, devido a sua importância em diversas áreas, notoriamente, nos ramos automobilístico e aeronáutico. Um dos processos que vem ganhando destaque é o método de sonificação, de forma alternativa à extração por meio de hidrólise ácida. As fibras oriundas de resíduos agroindustriais podem ser submetidas ao processo de sonificação em diferentes condições de temperatura, quantidades e tempos, as quais também podem ser funcionalizadas, a fim de seu uso como reforço de matrizes poliméricas. Neste trabalho, estudou-se as condições de sonificação em meio aquoso para obtenção de celulose nanoestruturada, com base no resíduo têxtil. Para essa finalidade, foram preparados compósitos com o uso do polietileno de baixa densidade como matriz 3, 6 e 9 % em massa de celulose nanoestruturada com alta cristalinidade, que foi obtida por sonificação ou celulose nanoestruturada com alta cristalinidade obtida por sonificação + poli (óxido de etileno), adicionando-se o óleo mineral puro, a fim de otimizar o processamento durante a moldagem. O poli (óxido de etileno) foi usado na proporção 1:10 em função da celulose nanoestruturada com alta cristalinidade obtida por sonificação + poli (óxido de etileno), para melhorar a interação fibra-matriz. Os resultados mostraram que foi possível a obtenção da celulose nanoestruturada, por meio do método de sonificação, que é um método sem a geração de resíduos. Portanto, a celulose nanoestruturada apresentou picos cristalinos característicos dos cristais polimórficos de uma mistura de celulose I e celulose II, evidenciando um valor de cristalinidade superior a celulose nanoestruturada obtida via química, que, por sua vez, demonstrou a importância do estudo das condições de otimização desse processo mecânico. Nessa direção, a análise por espectroscopia, na região do infravermelho, indicou que não há mudanças na estrutura química da celulose nanoestruturada, após a sonificação. A análise térmica mostrou estabilidade da celulose nanoestruturada sonificada superior à fibra sem tratamento e à fibra obtida via hidrólise química, indicando a possibilidade de uso como matriz em polímeros com temperaturas de fusão superiores às poliolefinas. Já a análise morfológica mostrou a redução nas dimensões da fibra e boa dispersão das fibras após ciclos de moldagem por compressão. Sendo assim, os resultados mecânicos de tração são comparáveis aos verificados na literatura para compósitos similares obtidos por processo de ix extrusão o qual permite uma melhor mistura e dispersão dos compósitos. A celulose nanoestruturada obtida mostrou ser interessante para aplicação em compósitos poliméricos. / At the present time, it has increased the debate and interest in cleaner production processes and economically viable nanostructured cellulose, due to its importance in several areas, notably in the automotive and aeronautical branches. One process that has been gaining importance is the sonication method, the extraction alternatively by acid hydrolysis. The fibers coming from agro-industrial waste can be subjected to sonification process in different conditions of temperature, amounts and times, which can also be functionalized, in order of their use as reinforcing polymeric matrices. In this study, we studied the conditions of sonication in an aqueous medium to obtain nanostructured cellulose-based textile residue. For this purpose, composites were prepared by using low density polyethylene as the matrix 3, 6 and 9 mass% of nanostructured cellulose with high crystallinity, which was obtained by sonication or nanostructured cellulose having high crystallinity obtained by sonification + poly (oxide ethylene), adding pure mineral oil in order to optimize the processing during molding. The poly (ethylene oxide) was used at 1:10 depending on the nanostructured cellulose having high crystallinity obtained by sonification + poly (ethylene oxide) to improve fiber-matrix interactions. The results showed that it was possible to obtain nanostructured cellulose, by means of sonication method, a method that is without generating waste. Therefore, the nanostructured cellulose was characteristic peaks of crystalline polymorphic crystals of a mixture of cellulose I and cellulose II, showing a higher value crystallinity cellulose obtained via the nanostructured chemical which, in turn, has demonstrated the importance of studying optimization conditions this mechanical process. In this sense, the analysis by spectroscopy in the infrared region, indicated that no changes in the structure of the nanostructured chemical pulp, after sonication. Thermal analysis showed stability of nanostructured sonicated above the pulp fiber and untreated fiber obtained via chemical hydrolysis, indicating the possibility of use as matrix polymers with melting temperatures higher than those polyolefins. Since the morphological analysis showed a reduction in fiber dimensions and good dispersion of the fibers after compression molding cycles. Thus, the mechanical traction results are comparable to those observed in the literature for similar composites obtained by the extrusion process which allows better mixing and dispersion of the composite. The nanostructured cellulose obtained proved to be interesting for application in polymeric composites.