Spontaneous co-speech hand gestures robustly affect learning and memory. Viewing or producing hand gestures during conversation facilitates the encoding, consolidation, and retention of the information in speech. Despite these effects, the cognitive and neural mechanisms supporting this relationship remains unknown. In Experiment 1, I explored the memory mechanisms supporting hand gesture by working with patients with damage to their hippocampus and thus their declarative memory system. Participants engaged in discourse tasks that disproportionately engaged the hippocampus. I found that patients gestured less overall than healthy comparisons across all tasks, suggesting that the hippocampus indeed plays a role in gesture production. In order to test whether non-declarative memory supports gesture production as well, Experiment 2 directly manipulated features of memory representations (both visual and motor) to determine what would guide the form of gesture when participants later explained their experiences. On three visits, amnesic patients, healthy comparison and brain-damaged comparison groups completed a Tower of Hanoi task, involving moving disks between pegs following a set of rules. On each visit, participants completed the task with different visual and motor information. Comparisons' gestures tended to reflect both visual and motor experience, while patients' gestures tended to rely more heavily on their motor experiences. This suggests that gesture may be supported by non-declarative memory as well, particularly in the absence of a declarative memory for what is being discussed. To directly test which properties of gesture facilitate learning, Experiment 3 examined how gesture affected the learning of novel labels for common, everyday objects. I again worked with patients with hippocampal amnesia, who are severely impaired in the learning of new words, along with healthy and brain-damaged comparisons. Participants were exposed to novel word-object pairing that either was learned with a gesture or not. For the gestured-with trials, the gesture was either viewed and then produced by the participant or passively viewed, allowing me to determine if production of a gesture was necessary for learning. After adequately learning all the word-object pairings, there was a 30-minute delay followed by a free recall and object identification task. Both comparison groups showed good learning of the words regardless of whether they were learned with gesture. The amnesic patients performed poorly on the recall task. On the object identification task, they were significantly more likely to identify the label-object pairing if the pairing had been learned with gesture. This benefit was only seen for those learned by producing gesture. For the pairings learned without gesture and the pairings learned with only viewing gesture, the patients were at chance. These findings demonstrate that gesture can help rescue hippocampal amnesics’ ability to bind labels with objects, and furthermore suggest that the self-production of gesture is critical for learning. These findings are the first to demonstrate a link between gesture and memory systems. Experiments 1 and 2 demonstrate that gesture can reflect information from both declarative and non-declarative memory. Experiment 3 demonstrates that the link between gesture and non-declarative memory can be exploited to facilitate learning in patients with memory impairment. By understanding how memory and language interact we will be able to exploit this interaction to benefit memory and language more generally.

The functional speech sound disorder, American English /r/ articulation errors, presents a unique and confounding clinical challenge as "therapy resistant" residual errors persist into adolescence and adulthood in many cases. Finding paucity of empirical research for /r/ treatment, evidence-based practice (EBP) exploration in motor-related disorders informed clinical practice and research directions. This study investigated the efficacy of "manual mimicry" (a kinesthetic, gestural, and visual cue) in treating intractable /r/ errors in a young adult using a single subject ABAB design. Perceptual accuracy judgments of three types of listeners (experts, graduate clinician, and na��ve listeners) indicated a positive treatment effect of manual mimicry cueing on vocalic /r/ sound productions. Electropalatograpy (EPG) outcome measures showed limited ability to accurately reflect perceptual changes quantitatively. These findings from an exploratory study provide initial evidence that perceptual saliency of /r/ productions may be potentially remediated using a kinesthetic, gestural, and visual cue during treatment. / Rangos School of Health Sciences / Speech-Language Pathology (SLP) / MS / Thesis

Interest in gesture production has considerably increased in recent decades, yet few studies have examined the preschool population. Even fewer studies have examined the intriguing interaction between motor skills and gesture. The original intent of this study was to investigate the relationship of gesture and motor skills in individuals who stutter. However due to recruitment limitations the enrolled sample consists solely of typically developing preschoolers, 3:8 to 6:6 years. Data are presented on gestures and disfluencies during spontaneous speech, a cartoon narration, and a video narration. Additionally, disfluencies were observed during a procedural description task with restricted hand use and hand tapping. Data indicated that higher frequencies of gestures and disfluencies were seen during the cartoon narration. A greater frequency of disfluencies was also experienced with restricted hand use. Relationships between the variables were also explored. Limitations and implications of these results are discussed from both theoretical and clinical perspectives. / Rangos School of Health Sciences / Speech-Language Pathology (SLP) / MS / Thesis

To have a voice link with other diving partners or surface personnel, divers need to put on a communication mask. The second stage regulator or mouthpiece is equipped with a circuit to pick up the voice of the diver. Then the voice is frequency-modulates into ultrasonic signal to be transmitted into water. A receiver on the other side picks up the ultrasonic signal and demodulates it back to voice, and plays back in diver's earphone set. This technology is mature but not widely adopted for its price. Most divers still use their favorite way to communicate with each other, i.e. DSL (divers' sign language.) As more and more intelligent machines or robots are built to help divers for their underwater task, divers not only need to exchange messages with their human partners but also machines. However, it seems that there are not many input devices available other than push buttons or joysticks. We know that divers¡¦hands are always busy with holding tools or gauges. Additional input devices will further complicate their movement, also distract their attention for safety measures. With this consideration, this paper intends to develop an algorithm to read the DSL as input commands for computer-aided diving system. To simplify the image processing part of the problem, we attach an LED at the tip of each finger. The gesture or the hand sign is then captured by a CCD camera. After thresholding, there will only five or less than five bright spots left in the image. The remaining part of the task is to design a classifier that can identify if the unknown sign is one from the pool. Furthermore, a constraint imposed is that the algorithm should work without knowing all of the signs in advance. This is an analogy to that human can recognize a face is someone known seen before or a stranger. We modify the concept of eigenfaces developed by Turk and Pentland into eigenhands. The idea is to choose geometrical properties of the bright spots (finger tips), like distance from fingertips to the centroid or the total area of the polygon with fingertips as its vertices as the features of the corresponding hand sign. All these features are quantitative, so we can put several features together to construct a vector to represent a specific hand sign. These vectors are treated as the raw data of the hand signs, and an essential subset or subspace can be spanned by the eigen vectors of the first few large corresponding values. It is less than the total number of hand signed involved. The projection of the raw vector along these eigen vectors are called the principal components of the hand sign. Principal components are abstract but they can serve as keys to match the candidate from a larger pool. With these types of simple geometrical features, the success rate of cross identification among 30 different subjects' 16 gestures varies to 91.04% .

Thesis (doctoral)--University of Surrey, 1993. / BLDSC reference no.: DX180487.

published_or_final_version / abstract / toc / Speech and Hearing Sciences / Master / Master of Science in Audiology

This thesis investigates cross-modal interactions in auditory and visual perception, focusing on the perception of expressive musical performances. A primary aim of the work is to advance knowledge pertaining to how and when musicians' body movements influence an observer's overall experience. Three studies, comprising two multi-factor experimental investigations and one theoretical contribution, explore the multi-modal experience of musical performance. The two empirical chapters investigate, respectively, (1) the real-time experience of musical structure and musical emotion, comparing unimodal and multimodal conditions, and (2) the multidimensional structure of affective responses to musical performance, as a function of sensory modalities and performance intentions. The theoretical chapter develops a class of quantitative models for studying real-time phenomena in music (in particular) and time-series data (in general). An original contribution of this thesis is to quantify the ways in which the auditory and visual components of musical performance contribute singly and in interaction with one another to overall experience. The studies show that seeing a musician performing can augment, complement and interact with the auditory component to significantly influence music perception. These results are relevant to, and inform theories on, multi-sensory integration, emotion, and music cognition, as well as performance practice and audio-video media.

This is an extensive study of the non-verbal in Genet's dramaturgy. Non-verbal forms constitute the plural, fragmented sum of theatrical possibility. Rhythms, movements, colours and shapes highlight the ritualised form of words and actions on and off stage. In Part One I define my understanding of Genet's theory of representation, and show how this theory informs his use of the non-verbal. On the one hand the discursive limits of Genet's reality forefront closure. On the other, within this closure an absence of transcendental meaning enables signs to be reconfigured and accorded a plurality of signification. A wealth of non-verbal scenic elements is codified and made to signify. But an antagonism between the triumphant liberation from inherent meaning and the inevitable falsity of representation underlies all Genet's theatre. Genet's reconfiguration involves transubstantiation, not substitution. It adds a supplementary layer of falsity to the sign. The co-presence of multiple layers of artifice effects a duality of belief and disbelief in the spectator, redefining the notion of theatricality. Non-verbal forms are of existential as well as theatrical import. Falsity is omnipresent. Genet thus destabilises and redramatises security, possession and identity. Part Two develops and illustrates the notion of the non-verbal elaborated in Part One through a predominantly stylistic study. I illustrate how performance on Genet's stage is a surface made of ritualised gestures and words, devoid of substance. Through constant polyphonic shifting characterisation is fragmented and unity of voice is denied. Central acoustic matrices are expanded forming homogenous blocks of repeated words, phonemes, stresses and prosodies. These blocks are juxtaposed with other rhythms creating chains of antagonistic structures that fracture traditional diegesis. Actors' gestures, tone, pitch, tempo and costume display a hybrid heterogeneity of styles which abolishes the monolithism of identity. The horizontal, vertical and diagonal lines constituting the set create a lattice network that fills a hypothetical vide with Genet's panoramic definition of reality. All these material signifiers resist metaphorical globalisation into themes or characters. They subsequently afford an opacity that fractures action into immediate acoustic and visual effects, and underscores form as surface detached from the oppressiveness of substance. And yet the absence of substance merely underscores the falsity of Genet's success. My concluding comments state that material, non-verbal artifice is freed from essentialist signification. It is therefore mobile, not static. The plural and liberated nature of the non-verbal enables Genet's singularity to be expressed, and in turn allows for the singularity of the spectator.

This study explores the meanings that participants in a British ELT setting give to teachers' non-verbal behaviours. It is a qualitative, descriptive study of the perceived functions that gestures and other non-verbal behaviours perform in the foreign language classroom, viewed mainly from the language learners' perspective. The thesis presents the stages of the research process, from the initial development of the research questions to the discussion of the research findings that summarise and discuss the participants' views. There are two distinct research phases presented in the thesis. The pilot study explores the perceptions of 18 experienced language learners of teachers' non-verbal behaviours. The data is collected in interviews based on videotaped extracts of classroom interaction, presented to the participants in two experimental conditions, with and without sound. The findings of this initial study justify the later change of method from the experimental design to a more exploratory framework. In the main study, 22 learners explain, in interviews based on stimulated recall, their perceptions on their teachers' verbal and non-verbal behaviours as occurring within the immediate classroom context. Finally, learners' views are complemented by 20 trainee teachers' written reports of classroom observation and their opinions expressed in focus group interviews. The data for the main study were thus collected through a combination of methods, ranging from classroom direct observations and videotaped recordings, to semi-structured interviews with language learners. The research findings indicate that participants generally believe that gestures and other non-verbal behaviours playa key role in the language learning and teaching process. Learners identify three types of functions that non-verbal behaviours play in the classroom interaction: (i) cognitive, i.e. non-verbal behaviours which work as enhancers of the learning processes, (ii) emotional, i.e. non-verbal behaviours that function as reliable communicative devices of teachers' emotions and attitudes and (iii) organisational, i.e. non-verbal behaviours which serve as tools of classroom management and control. The findings suggest that learners interpret teachers' non-verbal behaviours in a functional manner and use these messages and cues in their learning and social interaction with the teacher. The trainee teachers value in a similar manner the roles that non-verbal behaviours play in the language teaching and learning. However, they seem to prioritise the cognitive and managerial functions of teachers' non-verbal behaviours over the emotional ones and do not consider the latter as important as the learners did. This study is original in relation to previous studies of language classroom interaction in that it: • describes the kinds of teachers' behaviours which all teachers and learners are familiar with, but which have seldom been foregrounded in classroom-based research; • unlike previous studies of non-verbal behaviour, investigates the perceiver's view of the others' non-verbal behaviour rather than its production; • documents these processes of perception through an innovative methodology of data collection and analysis; • explores the teachers' non-verbal behaviours as perceived by the learners themselves, suggesting that their viewpoint can be one window on the reality of language classrooms; • provides explanations and functional interpretations for the many spontaneous and apparently unimportant actions that teachers use on a routine basis; • identifies a new area which needs consideration in any future research and pedagogy of language teaching and learning.

In this research, an approach is proposed for the robust tracking of upper body movement in unconstrained environments by using a Haar- Disparity algorithm together with a novel 2D silhouette projection algorithm. A cascade of boosted Haar classifiers is used to identify human faces in video images, where a disparity map is then used to establish the 3D locations of detected faces. Based on this information, anthropometric constraints are used to define a semi-spherical interaction space for upper body poses. This constrained region serves the purpose of pruning the search space as well as validating user poses. Haar-Disparity improves on the traditional skin manifold tracking by relaxing constraints on clothing, background and illumination. The 2D silhouette projection algorithm provides three orthogonal views of the 3D objects. This allows tracking of upper limbs to be performed in the 2D space as opposed to manipulating 3D noisy data directly. This thesis also proposes a complete optimal set of interactions for very large interactive displays. Experimental evaluation includes the performance of alternative camera positions and orientations, accuracy of pointing, direct manipulative gestures, flag semaphore emulation, and principal axes. As a minor part of this research interest, the usability of interacting using only arm gestures is also evaluated based on ISO 9241-9 standard. The results suggest that the proposed algorithm and optimal set of interactions are useful for interacting with large displays.