Das Ziel der vorliegenden Arbeit war es, die Mechanismen und die neuronale Implementierung des taktilen Arbeitsgedächtnisses (AG) zu erleuchten. Dazu wurde eine vibrotaktile Diskriminationsaufgabe mit den Methoden der funktionellen Magnetresonanztomografie (fMRT), Elektroenzephalografie (EEG), subliminalen elektrischen Stimulation und Psychophysik untersucht. Eine fMRT-Studie (Studie I) zeigte, dass ein ausgedehntes neuronales Netzwerk mit den einzelnen Aufgabenphasen, Enkodierung, Aufrechterhaltung, und Entscheidung, verbunden ist. Die Analyse oszillatorischer EEG-Aktivität (Studie II) und die selektive Inhibition des primären somatosensorischen Kortex (S1) durch subliminale Stimulation (Studie III) legen nahe, dass S1 nicht an der Aufrechterhaltung der vibrotaktilen Gedächtnisrepräsentation beteiligt ist. Das Ausmaß der S1-Aktivierung während der frühen Haltephase hängt vielmehr von der Effizienz, mit welcher Versuchspersonen den vibrotaktilen Reiz enkodieren, ab. Außerdem weisen Studie II and III darauf hin, dass bereits der physiologische Zustand von S1 in der Phase vor Beginn eines Versuchsdurchgangs eine einflussreiche Rolle spielt. Während Studie II zeigte, dass S1-Aktivität in dieser Phase, vermutlich durch attentionale top-down Modulation, unter AG-Anforderungen erhöht ist, ergab Studie III, dass eine optimierte bottom-up Verarbeitung in S1 zu einer verbesserten Aufgabenleistung führt. Verhaltensexperimente zeigten, dass Repräsentationen im Langzeitgedächtnis über die durchschnittliche Frequenz der präsentierten Vibrationsreize erheblich die Verhaltensleistung beeinflussen und den Zeitrichtungseffekt verursachen (Studie IV). Die Analyse der fMRT-Daten (Studie V) offenbarte außerdem, dass ein Netzwerk somatosensorischer Areale Informationen über die aktuelle Vibrationsfrequenz und die durchschnittliche Vibrationsfrequenz während der Enkodier- und Haltephase integriert. / The present dissertation aimed to shed more light on the psychological mechanisms and the neural basis of tactile working memory (WM). For this purpose, a vibrotactile delayed discrimination task was studied using the methods of functional magnetic resonance imaging (fMRI), electroencephalography (EEG), concurrent subliminal electrical stimulation and psychophysics. The fMRI study (Study I) showed that a broad network of brain regions - much broader than known from previous studies in non-human primates - supports the performance of a vibrotactile delayed discrimination task in the different task periods: encoding, maintenance, decision making. The analysis of oscillatory activity over the somatosensory cortex in the EEG study (Study II) and the experiment using subliminal electrical stimulation to locally inhibit the primary somatosensory cortex (S1) (Study III) suggest that S1 does not contribute to the active maintenance of the vibrotactile memory trace. The level of activity in S1 during the early delay period depends on the efficiency with which subjects encode the vibrotactile stimulus. Study II and III also showed that the activation level of S1 in the pre-trial period plays an important role. Study III suggests that, in this task period, S1 activity is up-regulated under WM demands probably reflecting the operation of top-down attentional control. Study III indicates that increasing local inhibition of S1 in the pre-trial period improves performance by facilitating bottom-up processing. Importantly, long-term memory representations of the average frequency of the stimulus set strongly influence performance giving rise to the time-order effect (Study IV). Additionally, the fMRI data (Study V) showed that a somatosensory network integrates information about the current vibrotactile stimulus and the representation of the average vibration frequency during stimulus encoding and maintenance.

Além de um canal para adquirir informações sobre o ambiente ao nosso redor, o sentido do tato é também o nosso sentido mais social. No entanto, a interação háptica é geralmente implementada como chamariz nas interfaces modernas. Embora a comunicação multimodal seja comum em Ambientes Virtuais, as tecnologias de Realidade Virtual mais acessíveis nem sequer incluem o componente háptico como parte fundamental. Esta tese apresenta estudos sobre percepção, desempenho do usuário, e experiência do usuário com dispositivos de comunicação vibrotátil construídos para suportar diferentes tarefas interativas em ambientes virtuais e físicos. Foram avaliados diferentes atuadores hápticos, configurações de exibição tátil, locais do corpo, perfis de usuário, e métodos para se projetar uma plataforma tátil robusta. Tal plataforma foi finalmente construída como uma tela vibrotátil para ser usada ao redor da cabeça e para suportar tarefas de consciência espacial e comunicação em ambientes virtuais e físicos. Durante a pesquisa foi observado que, apesar de sua importância para a comunicação, o uso proativo de háptica para intercomunicação é surpreendentemente negligenciado. Portanto, foi dada especial atenção aos elementos presentes na articulação da fala para introduzir a articulação háptica proativa como uma nova abordagem para intercomunicação. Foi proposto que a habilidade de usar uma interface háptica como uma ferramenta para comunicação implícita pode suplementar a comunicação e suportar tarefas colaborativas próximas e remotas em diferentes contextos. Além disso, uma interface articulatória pode fornecer um modo direto e expressivo de se comunicar através de sinais táteis. Para demonstrar isso, os resultados dessa pesquisa foram aplicados ao projeto de uma tela montada na cabeça com vibração, especialmente feita para interação com ambientes virtuais imersivos. Tal aparato mostrou-se útil não apenas para orientação no espaço 3D, mas também para intercomunicação em ambientes virtuais colaborativos. Além de nossas contribuições técnicas em relação à construção de uma tela tátil totalmente testada para múltiplas tarefas e contextos, nossa principal contribuição é a concepção e demonstração de um novo paradigma de interação tátil. Tal paradigma se concentra em fornecer maneiras simples e diretas para que indivíduos se expressem através de sinais táteis em aplicações mediadas por computador para interair com seu ambiente e com outros indivíduos. Esse paradigma envolve os usuários finais e permite que eles se tornem interlocutores ao invés de meros receptores do feedback tátil. / The sense of touch not only is a channel for acquiring information about the environment around us, it is also our most social sense. However, haptic interaction is usually implemented as a gimmick feature in modern interfaces. Although multimodal communication is commonplace in Virtual Environments, the most accessible Virtual Reality technologies do not even include the haptic component as a fundamental part. This thesis presents studies on perception, user performance, and user experience with vibrotactile communication devices built to support different interactive tasks in virtual and physical environments. We have assessed different haptic actuators, tactile display configurations, body sites, user profiles and methods to design a robust tactile platform. Such platform was finally built as a vibrotactile display to be worn around the head and to support spatial awareness and communication in both virtual and physical environments. During our research, we particularly notice that the proactive use of touch for intercommunication is surprisingly neglected regardless of its importance for communication. Therefore, we have also directed our attention to elements present in speech articulation to introduce proactive haptic articulation as a novel approach for intercommunication. We propose that the ability to use a haptic interface as a tool for implicit communication can supplement communication and support near and remote collaborative tasks in different contexts. In addition, an articulatory interface can provide a direct and expressive way for communicating through tactile cues. To demonstrate that, our results were applied to the design of a vibrotactile head-mounted display especially made for interaction with immersive virtual environments. Such apparatus was shown not only to support guidance in 3D space but also to support intercommunication in collaborative virtual environments. In addition to our technical contributions regarding the construction of a fully tested tactile display for multiple tasks and contexts, our main contribution is the conception and demonstration of a new paradigm for tactile interaction. Such paradigm focuses on providing simple and direct ways for individuals to express themselves through tactile cues in computer-mediated interaction with their environment and with others. Such paradigm embraces the final users and allows them to become interlocutors rather than just receivers of the haptic feedback.

Além de um canal para adquirir informações sobre o ambiente ao nosso redor, o sentido do tato é também o nosso sentido mais social. No entanto, a interação háptica é geralmente implementada como chamariz nas interfaces modernas. Embora a comunicação multimodal seja comum em Ambientes Virtuais, as tecnologias de Realidade Virtual mais acessíveis nem sequer incluem o componente háptico como parte fundamental. Esta tese apresenta estudos sobre percepção, desempenho do usuário, e experiência do usuário com dispositivos de comunicação vibrotátil construídos para suportar diferentes tarefas interativas em ambientes virtuais e físicos. Foram avaliados diferentes atuadores hápticos, configurações de exibição tátil, locais do corpo, perfis de usuário, e métodos para se projetar uma plataforma tátil robusta. Tal plataforma foi finalmente construída como uma tela vibrotátil para ser usada ao redor da cabeça e para suportar tarefas de consciência espacial e comunicação em ambientes virtuais e físicos. Durante a pesquisa foi observado que, apesar de sua importância para a comunicação, o uso proativo de háptica para intercomunicação é surpreendentemente negligenciado. Portanto, foi dada especial atenção aos elementos presentes na articulação da fala para introduzir a articulação háptica proativa como uma nova abordagem para intercomunicação. Foi proposto que a habilidade de usar uma interface háptica como uma ferramenta para comunicação implícita pode suplementar a comunicação e suportar tarefas colaborativas próximas e remotas em diferentes contextos. Além disso, uma interface articulatória pode fornecer um modo direto e expressivo de se comunicar através de sinais táteis. Para demonstrar isso, os resultados dessa pesquisa foram aplicados ao projeto de uma tela montada na cabeça com vibração, especialmente feita para interação com ambientes virtuais imersivos. Tal aparato mostrou-se útil não apenas para orientação no espaço 3D, mas também para intercomunicação em ambientes virtuais colaborativos. Além de nossas contribuições técnicas em relação à construção de uma tela tátil totalmente testada para múltiplas tarefas e contextos, nossa principal contribuição é a concepção e demonstração de um novo paradigma de interação tátil. Tal paradigma se concentra em fornecer maneiras simples e diretas para que indivíduos se expressem através de sinais táteis em aplicações mediadas por computador para interair com seu ambiente e com outros indivíduos. Esse paradigma envolve os usuários finais e permite que eles se tornem interlocutores ao invés de meros receptores do feedback tátil. / The sense of touch not only is a channel for acquiring information about the environment around us, it is also our most social sense. However, haptic interaction is usually implemented as a gimmick feature in modern interfaces. Although multimodal communication is commonplace in Virtual Environments, the most accessible Virtual Reality technologies do not even include the haptic component as a fundamental part. This thesis presents studies on perception, user performance, and user experience with vibrotactile communication devices built to support different interactive tasks in virtual and physical environments. We have assessed different haptic actuators, tactile display configurations, body sites, user profiles and methods to design a robust tactile platform. Such platform was finally built as a vibrotactile display to be worn around the head and to support spatial awareness and communication in both virtual and physical environments. During our research, we particularly notice that the proactive use of touch for intercommunication is surprisingly neglected regardless of its importance for communication. Therefore, we have also directed our attention to elements present in speech articulation to introduce proactive haptic articulation as a novel approach for intercommunication. We propose that the ability to use a haptic interface as a tool for implicit communication can supplement communication and support near and remote collaborative tasks in different contexts. In addition, an articulatory interface can provide a direct and expressive way for communicating through tactile cues. To demonstrate that, our results were applied to the design of a vibrotactile head-mounted display especially made for interaction with immersive virtual environments. Such apparatus was shown not only to support guidance in 3D space but also to support intercommunication in collaborative virtual environments. In addition to our technical contributions regarding the construction of a fully tested tactile display for multiple tasks and contexts, our main contribution is the conception and demonstration of a new paradigm for tactile interaction. Such paradigm focuses on providing simple and direct ways for individuals to express themselves through tactile cues in computer-mediated interaction with their environment and with others. Such paradigm embraces the final users and allows them to become interlocutors rather than just receivers of the haptic feedback.

Além de um canal para adquirir informações sobre o ambiente ao nosso redor, o sentido do tato é também o nosso sentido mais social. No entanto, a interação háptica é geralmente implementada como chamariz nas interfaces modernas. Embora a comunicação multimodal seja comum em Ambientes Virtuais, as tecnologias de Realidade Virtual mais acessíveis nem sequer incluem o componente háptico como parte fundamental. Esta tese apresenta estudos sobre percepção, desempenho do usuário, e experiência do usuário com dispositivos de comunicação vibrotátil construídos para suportar diferentes tarefas interativas em ambientes virtuais e físicos. Foram avaliados diferentes atuadores hápticos, configurações de exibição tátil, locais do corpo, perfis de usuário, e métodos para se projetar uma plataforma tátil robusta. Tal plataforma foi finalmente construída como uma tela vibrotátil para ser usada ao redor da cabeça e para suportar tarefas de consciência espacial e comunicação em ambientes virtuais e físicos. Durante a pesquisa foi observado que, apesar de sua importância para a comunicação, o uso proativo de háptica para intercomunicação é surpreendentemente negligenciado. Portanto, foi dada especial atenção aos elementos presentes na articulação da fala para introduzir a articulação háptica proativa como uma nova abordagem para intercomunicação. Foi proposto que a habilidade de usar uma interface háptica como uma ferramenta para comunicação implícita pode suplementar a comunicação e suportar tarefas colaborativas próximas e remotas em diferentes contextos. Além disso, uma interface articulatória pode fornecer um modo direto e expressivo de se comunicar através de sinais táteis. Para demonstrar isso, os resultados dessa pesquisa foram aplicados ao projeto de uma tela montada na cabeça com vibração, especialmente feita para interação com ambientes virtuais imersivos. Tal aparato mostrou-se útil não apenas para orientação no espaço 3D, mas também para intercomunicação em ambientes virtuais colaborativos. Além de nossas contribuições técnicas em relação à construção de uma tela tátil totalmente testada para múltiplas tarefas e contextos, nossa principal contribuição é a concepção e demonstração de um novo paradigma de interação tátil. Tal paradigma se concentra em fornecer maneiras simples e diretas para que indivíduos se expressem através de sinais táteis em aplicações mediadas por computador para interair com seu ambiente e com outros indivíduos. Esse paradigma envolve os usuários finais e permite que eles se tornem interlocutores ao invés de meros receptores do feedback tátil. / The sense of touch not only is a channel for acquiring information about the environment around us, it is also our most social sense. However, haptic interaction is usually implemented as a gimmick feature in modern interfaces. Although multimodal communication is commonplace in Virtual Environments, the most accessible Virtual Reality technologies do not even include the haptic component as a fundamental part. This thesis presents studies on perception, user performance, and user experience with vibrotactile communication devices built to support different interactive tasks in virtual and physical environments. We have assessed different haptic actuators, tactile display configurations, body sites, user profiles and methods to design a robust tactile platform. Such platform was finally built as a vibrotactile display to be worn around the head and to support spatial awareness and communication in both virtual and physical environments. During our research, we particularly notice that the proactive use of touch for intercommunication is surprisingly neglected regardless of its importance for communication. Therefore, we have also directed our attention to elements present in speech articulation to introduce proactive haptic articulation as a novel approach for intercommunication. We propose that the ability to use a haptic interface as a tool for implicit communication can supplement communication and support near and remote collaborative tasks in different contexts. In addition, an articulatory interface can provide a direct and expressive way for communicating through tactile cues. To demonstrate that, our results were applied to the design of a vibrotactile head-mounted display especially made for interaction with immersive virtual environments. Such apparatus was shown not only to support guidance in 3D space but also to support intercommunication in collaborative virtual environments. In addition to our technical contributions regarding the construction of a fully tested tactile display for multiple tasks and contexts, our main contribution is the conception and demonstration of a new paradigm for tactile interaction. Such paradigm focuses on providing simple and direct ways for individuals to express themselves through tactile cues in computer-mediated interaction with their environment and with others. Such paradigm embraces the final users and allows them to become interlocutors rather than just receivers of the haptic feedback.

There are different ways to interact with different hardware, therefore it is important to have an understanding about what factors that affect the experience when designing interactions and interfaces. This study focuses on exploring how auditory and vibrotactile feedback are perceived by the users when they interact in a virtual AR environment. An application was developed to the AR glasses Magic Leap with different interactions, both passive and active. An experimental study was conducted with 28 participants that got to interact in this virtual environment. The study included two parts. First the participants interacted in the virtual environment where they did a think aloud. Thereafter they were interviewed. There were a total of three test cases. One with only auditory feedback, one with vibrotactile feedback, and a third that had both auditory and vibrotactile feedback. Seven of the 28 participants acted as a control group that did not have any feedback to their interactions. The study shows that using only vibrotactile feedback creates different impressions depending on earlier experiences with the same AR environment. Using only auditory feedback created an atmosphere that were close to reality. Having both feedbacks active at the same time reduced the noticed feedback and some interactions were here not even noticed at all. Passive interactions were more noticed than active interactions in all cases. / Det finns flera olika sätt att interagera med olika hårdvaror och därför är det viktigt att ha en förståelse kring vilka faktorer som påverkar upplevelsen när man designar för diverse gränssnitt och interaktioner. Den här studien fokuserar på att utforska hur auditiv och vibrationsåterkoppling uppfattas av användaren när de interagerar i en virtuell AR-miljö. En applikation var utvecklad till AR-glasögonen Magic Leap One med olika aktiva och passiva interaktioner.En experimentell studie genomfördes med 28 deltagare som fick interagera i en virtuell miljö. Studien bestod av av två delar. Först fick deltagarna interagera i en virtuell miljö där de gjorde en think aloud. Efter detta blev de intervjuade. Det var totalt tre testfall, ett hade endast auditiv återkoppling, ett hade vibrationsåterkoppling och det sista hade både auditiv och vibrationsåterkoppling. Sju av de 28 deltagarna agerade kontrollgrupp och de hade ingen återkoppling på deras interaktioner.Studien visade att bara använda vibrationsåterkoppling skapade olika intryck beroende på de tidigare erfarenheterna i samma AR-miljö. Att endast använda auditiv återkoppling skapade en atmosfär som vara nära verkligheten. Att ha båda återkopplingarna aktiva samtidigt reducerade den totala märkta återkopplingen och några interaktioner hade inte någon person som noterade någon av dem. Passiva interaktioner var mer uppmärksammade än aktiva interaktioner i alla testfallen.

<p>Our study investigates how different mapping strategies of distance information affect performance in an object exploration task with a teleoperated virtual robot. The task was to find an object inside a backpack using a simulated robotic gripper. A virtual proximity sensor tracked the distance between the tip of the gripper and the object. The distance was conveyed as a vibration pattern on the users index finger. This is the only information that was received to guide the user towards the object. The goal was to locate the hidden object by moving the tip of the gripper as quickly and as closely towards the object as possible without touching it. We implemented three different mapping strategies that utilized continuous frequency and amplitude variations of sinusoidal vibrations to encode distance. The present study provides empirical evidence that the mapping strategy can affect accuracy when approaching an object. We found that linear feedback sensations help to sense the rate of approach. Non- linear feedback perception can provide cues that enable more accurate approximation of the absolute distance. We found that experienced participants could selectively attend to and integrate frequency and intensity cues when both modalities are changed simultaneously. Inexperienced participants were not able to make this distinction and found it difficult to interpret such a signal. They preferred one-dimensional changes.</p>

No description available.

abstract: One type of assistive device for the blind has attempted to convert visual information into information that can be perceived through another sense, such as touch or hearing. A vibrotactile haptic display assistive device consists of an array of vibrating elements placed against the skin, allowing the blind individual to receive visual information through touch. However, these approaches have two significant technical challenges: large vibration element size and the number of microcontroller pins required for vibration control, both causing excessively low resolution of the device. Here, I propose and investigate a type of high-resolution vibrotactile haptic display which overcomes these challenges by utilizing a ‘microbeam’ as the vibrating element. These microbeams can then be actuated using only one microcontroller pin connected to a speaker or surface transducer. This approach could solve the low-resolution problem currently present in all haptic displays. In this paper, the results of an investigation into the manufacturability of such a device, simulation of the vibrational characteristics, and prototyping and experimental validation of the device concept are presented. The possible reasons of the frequency shift between the result of the forced or free response of beams and the frequency calculated based on a lumped mass approximation are investigated. It is found that one of the important reasons for the frequency shift is the size effect, the dependency of the elastic modulus on the size and kind of material. This size effect on A2 tool steel for Micro-Meso scale cantilever beams for the proposed system is investigated. / Dissertation/Thesis / Doctoral Dissertation Systems Engineering 2019

<p>In this thesis we study algorithms and architectures that can provide a better Quality of Experience (QoE) for streaming video systems and services. With cases and examples taken from the application scenarios of football on mobile phones, we address the fundamental problems behind streaming video services. Thus, our research results can be applied and extended to other networks, to other sports and to other cultural activities.</p><p>In algorithm development, we propose five different schemes. We suggest a blind motion estimation and a trellis based motion estimation with dynamic programming algorithms for Wyner-Ziv coding. We develop a trans-media technology, vibrotactile coding of visual signals for mobile phones. We propose a new bandwidth prediction scheme for real-time video conference. We also provide an effective method based on dynamic programming to select optimal services and maximize QoE.</p><p>In architecture design, we offer three architectures for real-time interactive video and two for streaming live football information. The former three are: a structure of motion estimation in Wyner-Ziv coding for real-time video; a variable bit rate Wyner-Ziv video coding structure based on multi-view camera array; and a dynamic resource allocation structure based on 3-D object motion. The latter two are: a vibrotactile signal rendering system for live information; and a Universal Multimedia Access architecture for streaming live football video.</p><p>In QoE exploration, we give a detailed and deep discussion of QoE and the enabling techniques. We also develop a conceptual model for QoE. Moreover we place streaming video services in a framework of QoE. The new general framework of streaming video services allows for the interaction between the user, content and technology.</p><p>We demonstrate that it is possible to develop algorithms and architectures that take into account the user's perspective. Quality of Experience in video mobile services is within our reach.</p>

In this thesis we study algorithms and architectures that can provide a better Quality of Experience (QoE) for streaming video systems and services. With cases and examples taken from the application scenarios of football on mobile phones, we address the fundamental problems behind streaming video services. Thus, our research results can be applied and extended to other networks, to other sports and to other cultural activities. In algorithm development, we propose five different schemes. We suggest a blind motion estimation and a trellis based motion estimation with dynamic programming algorithms for Wyner-Ziv coding. We develop a trans-media technology, vibrotactile coding of visual signals for mobile phones. We propose a new bandwidth prediction scheme for real-time video conference. We also provide an effective method based on dynamic programming to select optimal services and maximize QoE. In architecture design, we offer three architectures for real-time interactive video and two for streaming live football information. The former three are: a structure of motion estimation in Wyner-Ziv coding for real-time video; a variable bit rate Wyner-Ziv video coding structure based on multi-view camera array; and a dynamic resource allocation structure based on 3-D object motion. The latter two are: a vibrotactile signal rendering system for live information; and a Universal Multimedia Access architecture for streaming live football video. In QoE exploration, we give a detailed and deep discussion of QoE and the enabling techniques. We also develop a conceptual model for QoE. Moreover we place streaming video services in a framework of QoE. The new general framework of streaming video services allows for the interaction between the user, content and technology. We demonstrate that it is possible to develop algorithms and architectures that take into account the user's perspective. Quality of Experience in video mobile services is within our reach.