Yes / Online retailers are increasingly using augmented reality (AR) and virtual reality (VR) technologies to solve mental and physical intangibility issues in a product evaluation. Moreover, the technologies are easily available and accessible to consumers via their smartphones. The authors conducted three experiments to examine consumer responses to technology interfaces (AR/VR and mobile apps) for hedonic and utilitarian products. The results show that AR is easier to use (vs. app), and users find AR more responsive when buying a hedonic (vs. utilitarian) product. Touch interface users are likely to have a more satisfying experience and greater recommendation intentions, as compared to AR, for buying utilitarian products. In contrast, a multisensory environment (AR) results in a better user experience for purchasing a hedonic product. Moreover, multisensory technologies lead to higher visual appeal, emotional appeal, and purchase intentions. The research contributes to the literature on computer-mediated interactions in a multisensory environment and proposes actionable recommendations to online marketers.

Yes / Our motor actions normally generate sensory events, but how do we know which events were self generated and which have external causes? Here we use temporal adaptation to investigate the processing stage and generality of our sensorimotor timing estimates. Methodology/Principal Findings: Adaptation to artificially-induced delays between action and event can produce a startling percept¿upon removal of the delay it feels as if the sensory event precedes its causative action. This temporal recalibration of action and event occurs in a quantitatively similar manner across the sensory modalities. Critically, it is robust to the replacement of one sense during the adaptation phase with another sense during the test judgment. Conclusions/Significance: Our findings suggest a high-level, supramodal recalibration mechanism. The effects are well described by a simple model which attempts to preserve the expected synchrony between action and event, but only when causality indicates it is reasonable to do so. We further demonstrate that this model successfully characterises related adaptation data from outside the sensorimotor domain.

No description available.

The experiments reported within this thesis use psychophysical techniques to examine the factors which determine perceived multisensory timing in humans. Chapters 1 and 2 describe anatomical and psychophysical features of temporal processing, respectively, whilst Chapter 3 introduces the reader to psychophysical methods. Chapter 4 examines the relationship between two measures of sensory latency, reaction time (RT) and crossmodal temporal order judgment (TOJ). Despite task and attentional manipulations the two measures do not correlate, suggesting that they measure some fundamentally different aspect(s) of temporal perception. Chapter 5 examines the effects of adaptation to asynchronous stimulus pairs on perceived audiovisual (AV), audiotactile (AT) and visuotactile (VT) temporal order. Significant temporal shifts are recorded in all three conditions. Evidence is also presented showing that crossmodal TOJs are intransitive. Chapter 6 shows that concurrent adaptation to two sets of asynchronous AV stimulus pairs causes perceived AV temporal order to recalibrate at two locations simultaneously, and that AV asynchrony adaptation effects are significantly affected by observers' attention during adaptation. Finally, Chapter 7 shows that when observers are accustomed to a physical delay between motor actions and sensory events, an event presented at a reduced delay appears to precede the causative motor action. The data are well-described by a simple model based on a strong prior assumption of physical synchrony between motor actions and their sensory consequences.

Titel: Se, hör & shoppa! En undersökning kring hur det med visuell och auditiv stimuli går att påverka kundens rörelsemönster och köpbeteende i butik. Författare: Klara Johansson, Marina Johansson Handledare: Professor Bertil Hultén Examinator: Universitetslektor Leif V Rytting Kurs: Företagsekonomi III – Examensarbete inriktning detaljhandel och service management (kandidat) 15hp Problemformulering: Hur påverkas kundens rörelsemönster och köpbeteende i en butiksmiljö med hjälp av visuella stimuli samt auditivt stimuli? Syfte: Syftet med examensarbetet är att undersöka, analysera och klargöra huruvida kundens rörelsemönster och köpbeteende kan påverkas med hjälp av sensoriska stimuli. Vi vill med hjälp av en multisensorisk upplevelse, vita pilar samt ljudet av fågelkvitter, försöka påverka kundens vägval i butiken för att bidra till att denne ska uppmärksamma större del av butiksytan. Metod: Examensarbetet har utformats med ett deduktivt angreppssätt. Vi har gjort en kvantitativ analys med en experimentell ansats. En omfattande litteraturstudie har legat till grund för de hypoteser som formulerats, i syfte att svara på forskningsfrågan. Primärdata har samlats in genom sammanlagt 1347 observationer på Plantagen i Kalmar. Slutsats: En slutsats av arbetet är att kunderna påverkas till att följa visuella stimuli i form av pilar. Det är därmed möjligt att påverka kunden att se mer av butikens yta med hjälp av visuella stimuli. Vi kan även visa att tillsättning av auditivt stimuli och visuella stimuli påverkar köpbeteendet i form av tid spenderad i experimentzonen, men det behövs vidare forskning för att utreda i vilken grad påverkan sker. Nyckelord: Sinnesmarknadsföring, visuell stimuli, auditiv stimuli, butiker, rörelsemönster, multisensorik, kongruens / Title: See, hear & shop! A Study of how visual and auditory stimuli can influence the customer's movement patterns and buying behaviour in store. Authors: Klara Johansson, Marina Johansson Supervisor: Professor Bertil Hultén Ph.D. Examiner: Senior Lecturer Leif V Rytting Ph.D. Course: Business economies III – Retail and service management (C level) Issues: How are the customer's movement patterns and buying behaviour affected in store with the help of visual and auditory stimuli? Purpose: The purpose of this study is to investigate, analyse and clarify whether the customer's movement patterns and buying behaviour can be affected by means of sensory stimuli. With the help of sensory marketing, white arrows and bird song, we want to influence the customer's choice of direction in the store with the purpose of making the customer notice a wider part of the store. Methodology: The thesis has been designed with a deductive approach. We have done a quantitative analysis with an experimental approach. An extensive search for literature has been the basis of the hypotheses formulated, in order to answer the research question. Primary data were collected through a total of 1347 observations at Plantagen in Kalmar. Findings: A finding of this thesis is that customers are influenced to follow the visual stimuli by the means of white arrows. It is thereby possible to affect customers to notice a wider part of the store by the means of visual stimuli. We can also show that adding auditory stimuli and visual stimuli can affect the buying behaviour in the matter of time spent in the experimental zone, although further research is required to exam to which extent the impact occurs. Keywords: Sensory marketing, visual stimuli, auditory stimuli, in stores, movement patterns, multisensory, congruence

Notre travail s’inscrit dans une conception fonctionnaliste de la mémoire qui se donne pour objectif de décrire le fonctionnement de la mémoire plus que sa structure. Notre assise théorique est basée sur le modèle Act-in qui postule que la mémoire est représentée par un ensemble de traces multidimensionnelles. Ces traces comporteraient l’ensemble des dimensions d’une expérience (e.g., motrices, sensorielles, émotionnelles). Pour rendre compte de l’émergence des connaissances, ce modèle repose sur deux mécanismes : l’activation et l’intégration. Notre travail avait pour objectif de spécifier les relations fonctionnelles entre activation et intégration à travers les mécanismes perceptifs et les mécanismes mnésiques. Nous avons également proposé un mécanisme d’interaction, permettant de rendre compte de la formation des traces sensorielles en mémoire. Comme le modèle Act-in conçoit les traces en mémoire comme étant multidimensionnelles, nous avons étudié les différentes relations entre mécanismes d’intégration, interaction et activation via le prisme de la multimodalité. Ce travail s’est organisé autour de deux grands axes expérimentaux, un premier tourné vers l’intégration en perception et l’autre vers l’intégration en mémoire.Dans un premier axe, nous avons introduit la notion d’interaction multimodale pour rendre compte des relations entre modalités sensorielles dans des tâches perceptives. Nous avons montré que lorsque des interactions sensorielles étaient systématiquement répétées, les performances de sujets étaient meilleures que celles observées dans un contexte multimodal dans lequel les modalités sensorielles n’entretenaient pas de lien. Nous avons mesuré à l’aide d’un outil mathématique, le Race Model, le gain multi-sensoriel dans diverses conditions. Nous avons montré que le Race Model mesurait un gain multi-sensoriel essentiellement lorsque la tâche n’impliquait pas de processus mnésiques (i.e., détection). Selon nous, cela montre que cet outil mathématique est plus sensible à l’interaction multimodale, lorsque les composants sensoriels ne sont pas intégrés en mémoire.Dans un second axe, nous avons développé un paradigme expérimental permettant de mesurer les effets d’un encodage sensoriel sur une récupération ultérieure (i.e., catégorisation, reconnaissance et rappel libre). Nous avons montré qu’une association multimodale congruente lors d’un encodage (e.g., image de chien et aboiement du chien) améliorait significativement les performances dans diverses tâches de mémoire. Selon nous, l’activation des dimensions auditives associées au sein de la trace explique cette facilitation lors de la récupération. Nous avons également montré qu’une association multimodale non congruente lors de l’encodage (e.g., image d’un chien et bruit blanc) perturbait la récupération en mémoire à long terme. Ce résultat apporte un argument supplémentaire en faveur des modèles de mémoire qui postulent que les connaissances sont avant tout sensorielles.Pour résumer, ce travail a permis dans un premier temps de proposer un mécanisme d’interaction multimodale nécessaire à l’intégration des composants sensoriels en mémoire. Dans un second temps, nous avons montré que des modalités sensorielles associées lors de l’encodage pouvait être réactivées ultérieurement et faciliter le traitement uniquement si les modalités sensorielles étaient sémantiquement congruentes. Ce résultat renforce la conception d’une mémoire organisée autour de connaissances avant tout sensorielles.Enfin, les résultats que nous obtenons suggèrent que la mémoire et la perception partagent des traitements communs car un trait perceptivement absent peut influencer un trait perceptivement présent et réciproquement. Cela nous permet d’envisager une relation horizontale entre mémoire et perception et remet en question la conception traditionnelle top-down des liens entre mémoire et perception. / Our conception of memory has the aim to describe the memory process more than the memory architecture. Our theoretical approach is based on the Act-in model, postulating that memory is represented by a multidimensional set of traces. These traces should include all dimensions of an experience (e.g., sensory-motor and emotional components). This model relies on two mechanisms (i.e., activation and integration) in order to reveal the emergence of knowledge. The aim of this work was to specify the functional relationships between activation and integration through memory and perception mechanisms. We also proposed a mechanism of interaction, to understand the formation of sensory memory traces. The model Act-in is composed by multidimensional memory traces. Thus, we studied the relationship between integration, interaction and activation through multimodality. This work was organized around two main axis. On one hand perceptual integration and on the other hand the memory integration.In the first axis, we have introduced the concept of multimodal interaction to show the relationship between sensory modalities in perceptual tasks. Our results revealed that when sensory interactions were systematically repeated, the performance of subjects were better than those observed in a context in which multimodal sensory modalities did not maintain a link. We measured using a mathematical tool, the Race Model, multi-sensory gain under various conditions. We have shown that the Race Model measured multi-sensory gain, mainly when the task did not involve memory processes (i.e., detection). This mathematical tool seems to be more sensitive to multimodal interaction, when the sensory components are not integrated into memory.In the second axis, we used an experimental paradigm to measure the effects of sensory encoding on subsequent retrieval tasks (i.e., categorization, recognition and free recall). Our results showed that a congruent multimodal association at encoding (e.g., image of dog and the dog barking) improved performance in various memory tasks. It seems that the activation of the sensory dimensions could explain the facilitation during retrieval. Moreover, an incongruent multimodal association at encoding (e.g., image of a dog and white noise) disturbed performance from retrieval in long-term memory. This result provides a further argument supporting sensory-based memory model.In summary, we propose a multimodal interaction mechanism for the integration of sensory components in memory. Secondly, we have shown that sensory modalities associated during encoding could be reactivated later and facilitate the processing only when the sensory modalities were semantically congruent. This result improves the assumption that memory is sensory-based.Finally, theses results suggest that memory and perception share common features. The reactivation of a sensory dimension can influence the perceptual dimension processing and vice versa. This allows us to consider a horizontal relationship between memory and perception. In such case, the traditional relationship between memory and perception, as assumed to be top-down, could be questioned.

A integração audiovisual é capaz de melhorar a performance e a aprendizagem unissensorial. Esta integração ocorre mesmo quando uma das informações sensoriais não é consciente ao indivíduo, e.g. informação semântica auditiva pode impactar percepção visual inconsciente. No âmbito da percepção visual não consciente, o fluxo da informação fica restrito ao início do processamento cortical, não alcançando áreas de alta ordem, como a rede parietal-frontal. Considerando que as interações multissensoriais no córtex podem ocorrer desde estágios iniciais do processamento, levantamos a hipótese que a apresentação de estímulos visuais não conscientes pode beneficiar a aprendizagem auditiva de percepção de notas musicais. Neste estudo, testamos sujeitos em um paradigma de aprendizagem auditiva. Os indivíduos tinham como tarefa identificar seis notas diferentes separadas por 50 cents, ou ¼ de tom entre si. Os grupos foram divididos de acordo com o treinamento: auditivo (A), Auditivo + visual não consciente congruente (AV), e Auditivo + visual não consciente incongruente (AVi). Eles passaram por testes pré- e pós-treinamento de escolha forçada dentre seis alternativas somente com informação auditiva. Registramos dados de Eletroencefalografia (EEG) ao longo do experimento. A performance foi calculada por meio do Desvio Médio Absoluto das respostas com relação às notas corretas. Resultados comportamentais mostraram que o grupo AV não somente teve uma melhor performance durante o treinamento, quando a informação visual não consciente estava presente, como também apresentou um maior efeito do treinamento (melhora de pré- para pós-treinamento). Ambos os controles não diferiram entre si. Também mostramos que estes efeitos são devidos às notas da extremidade da escala. Os resultados de EEG mostram que a banda theta tem um papel fundamental na melhora da performance através do aumento da sincronização em áreas centrais e temporais, e da conectividade funcional entre córtices visuais e auditivos. Estes resultados são uma demonstração que asinterações multissensoriais ocorridas sem consciente são fortes o suficiente para aumentar a aprendizagem auditiva. Concluimos também que a conectividade funcional entre áreas corticais de processamento inicial aumenta depois de poucos minutos de treinamento. Além disso, esta tese levanta novas possibilidades para protocolos de aprendizagem perceptual auditiva, especialmente aqueles aplicado para aprendizagem de reconhecimento de notas musicais / Audiovisual integration may improve unisensory perceptual performance and learning. Interestingly, this integration may occur even when one of the sensory modalities is not conscious to the subject, e.g., semantic auditory information may impact nonconscious visual perception. Studies have shown that the flow of nonconscious visual information is mostly restricted to early cortical processing, without reaching higher-order areas, such as the parieto-frontal network. Thus, because multisensory cortical interactions may already occur in early stages of processing, we hypothesized that nonconscious visual stimulation without semantic information might facilitate auditory pitch learning. In this study we used a pitch learning paradigm, in which individuals had to identify six pitches in a scale with constant intervals of 50 cents. Subjects were assigned to one of three training groups: the test group (Auditory + congruent unconscious visual, AV), and two control groups (Auditory only, A, and Auditory + incongruent unconscious visual, AVi). Auditory-only tests were done before and after training in all groups. Electroencephalography (EEG) was recorded throughout the experiment. Results show that the test group (AV, with congruent nonconscious visual stimuli) performed better during the training, and showed a greater improvement from pre- to post-test. Control groups did not differ from one another. Changes in the AV group were mainly due to performances in the first and last pitches of the scale. We also observed consistent EEG patterns associated with this performance improvement in the AV group, especially higher theta-band synchronization among central and temporal areas, and stronger theta-band functional connectivity between visual and auditory cortices. Therefore, we show that nonconscious multisensory interactions are powerful enough to boost auditory perceptual learning, and that increased functional connectivity between early visual and auditory cortices after training might play a role in this effect. Moreover, weprovide a methodological contribution for future studies on auditory perceptual learning, particularly those applied to relative and absolute pitch training

Herta Müller represents physical suffering and repression in her works, often reflecting on the regime of Nicolae Ceaușescu, and her constant interest in language and reflexivity towards writing have led her to develop sophisticated metaphors that she uses to illuminate language and its functioning under such subjugation. With reference to her fiction and non-fiction, I demonstrate how she uses concrete ideas to understand linguistic phenomena. She evokes injury, destruction, force, life, space, touch, silence, and other bodily experiences to make sense of language in the condition of suffering from social oppression. Drawing on conceptual metaphor theory within the framework of cognitive literary studies, I argue that Müller both relies on and estranges the ways in which people speak and think about language. Language is imagined differently depending on the circumstances and in close relationship with various sensory experiences. The complexity of the relationship between language and thought problematises the process of metaphor building and makes it difficult to identify its key aspects across different contexts and sensory modalities. Müller’s tropes are easy to experience, but difficult to analyse. The idea of language does not exist as a stable concept and is regularly reimagined in her texts; but its meaning is not arbitrary and depends on bodily experience. While Müller evokes such experience to understand language in the condition of suffering, she can also use linguistic concepts to elucidate more abstract ideas. Language can be regarded as an abstract or concrete phenomenon depending on the relevant bodily, linguistic, and cultural contexts. This project contributes to the study of Müller’s poetics as well as to the literary critical interpretation of embodied cognition, and develops the use of conceptual metaphor theory for literary analysis. It also seeks to develop understanding of the role of bodily experience in the metaphorical conceptualisation of language.

La rééducation fonctionnelle classique comprend des séances thérapeutiques, des exercices à domicile et des mouvements avec ou sans l'aide de thérapeutes. Cette approche classique souffre de nombreuses limitations, en raison de l’incapacité de l’expert à suivre les séances à domicile du patient et du manque de motivation du patient pour répéter les exercices sans retour. Les jeux sérieux ont été présentés comme une solution à ces problèmes. Premièrement, des exergames ont été développés à l’aide d’une méthodologie de codesign, où patients, experts et développeurs ont pris part aux procédures de conception. Le capteur Kinect a été utilisé pour capturer la cinématique en temps réel au cours de l’exercice. Ensuite, une fusion de données a été étudier entre le capteur Kinect et les unités de mesure inertielles, afin d’augmenter la précision de l’estimation de l’angle des articulations, à l’aide d’une approche système de systèmes. Le système a été évalué par des patients avec différentes pathologies au cours de multiples campagnes d'évaluation. Les résultats obtenus montrent que les jeux sérieux peuvent être une solution pour des pathologies spécifiques. En outre, les experts ont été convaincus de la pertinence clinique de ce dispositif et ont estimé que les données proposées étaient suffisantes pour évaluer la situation du patient. La mise en place de tels systèmes maximiserait l’efficacité du programme de réadaptation. D'autre part, ce système permettrait également de réduire les limitations actuellement présentes dans les programmes de rééducation classiques, permettant aux patients de visualiser leurs mouvements et aux experts de suivre l'exécution de l'exercice à domicile. / Conventional musculoskeletal rehabilitation consists of therapeutic sessions, home exercise assignment, and movement execution with or without the assistance of therapists. This classical approach suffers from many limitations, due to the expert’s inability to follow the patient’s home sessions, and the patient’s lack of motivation to repeat the same exercises without feedback. Serious games have been presented as a possible solution for these problems. This thesis was carried out in the eBioMed experimental platform of the Université de technologie de Compiège, and in the framework of the Labex MS2T. The aim of this thesis is to develop a real-time, serious gaming system for home-based musculoskeletal rehabilitation. First, exergames were developed, using a codesign methodology, where the patients, experts and developers took part in the design and implementation procedures. The Kinect sensor was used to capture real-time kinematics during each exercise. Next, data fusion was implemented between the Kinect sensor and inertial measurement units, to increase the accuracy of joint angle estimation, using a system of systems approach. In addition, graphical user interfaces were developed, for experts and patients, to suit the needs of different end-users, based on the results of an end-user acceptability study. The system was evaluated by patients with different pathologies through multiple evaluation campaigns. Obtained results showed that serious games can be a good solution for specific types of pathologies. Moreover, experts were convinced of the clinical relevance of this device, and found that the estimated data was more than enough to assess the patient’s situation during their home-based exercise sessions. Finally, during these three years, we have set the base for a home-based rehabilitation system that can be deployed at home or in a clinical environment. The implementation of such systems would maximize the efficiency of rehabilitation program, while saving the patient’s and expert’s time and money. On the other hand, this system would also reduce the limitation that are currently present in classical rehabilitation programs, allowing the patients to visualize their movements, and the experts to follow the home exercise execution.

The classic distinction between the body schema and the body image received renewed interest in cognitive psychology, in part because of the attempts by the leading psychologist Charles Spence and his co-authors to synthesise a mounting body of research into the multisensory nature and functional properties of the neural structures in primate cortex that are sensitive and responsive to cross-modal stimuli generated from the body and objects located close to the body, and the famous rubber hand illusion which purported to illustrate how the perception and understanding of what counts as one’s body, i.e., our body image, can be manipulated to include foreign, body-part-like, objects such as a rubber hand. This approach was intended to settle age old questions about how the body schema – the system sub-personal sensorimotor system that shapes, facilitates and regulates motor control – is implemented in the brain and address historic confusions about how the body schema should be understood as an explanatory concept, as well as the problems surrounding the body schema and image distinction on the grounds of the persistent conflation between the two concepts. However, after offering several proposals as to how the body schema should be used to organise and interpret the empirical data, the distinction fell out of favour with Spence and his colleagues on the grounds of the very problems they intended to resolve. The proposed solution is an alternative theoretical framework that, I shall argue, never materialised. Instead, the various definitions they disseminate, I will claim, simply serve to further perpetuate the same problems and confusions about the body schema. Thus, the current state of the literature on the body image and schema in cognitive psychology is in dire need of a conceptual framework that would help us situate and interpret the important empirical data. I propose that we revisit the philosophical debates that were inspired by the philosopher Shaun Gallagher as part of his project to provide a conceptual analysis of the body schema and image distinction and vindicate its status as an important explanatory device for the explanatory ambitions of embodied cognition. Gallagher’s analysis opens up important questions about how the sub-personal multisensory processes of the body schema not only facilitate moment-by-moment motor behaviours, but how they shape and optimise motor control across developmental timelines, as well the importance of the embodied configuration of an agent and its particular eco-niche for shaping and facilitating its motor behaviours. The second important argument of the thesis is that the response to Gallagher’s analysis has simply served to suppress the line of research that Gallagher inspired because the questions his analysis raises have been overshadowed by more general disputes between Gallagher and his opponents about the shape an analysis of the body schema from the perspective of embodied cognition should take. As such, potentially promising lines of research in relation to the body schema have since dried up. As part of my attempt to make progress on the issues that are laid out at the first and second stages of the thesis, the third stage will involve an exploration into the seminal Bayesian approach to understanding cross-modal cue optimisation as it applies to object perception (Banks & Ernst, 2002) and the recent extension of this paradigm to the multimodal sensorimotor processes that underpin motor behaviour in action-oriented cognitive science (e.g., Friston, 2010). The conclusion of the thesis is that the move from an embodied to an action-oriented analysis of the body schema, and the conceptual distinction of which it is part, provides us with the right kind of theoretical resources to begin to pursue fruitful avenues of research that allow us to begin to address the questions set out by Gallagher’s analysis whilst avoiding (some of) the pitfalls that beset the embodied approach. In the final chapter I use this model of the body schema to illustrate how it can provide the basis for working back up towards a comprehensive theory of the body image and schema distinction, which I then bring to bear on current, as-yet-unaddressed, issues in developmental psychology.