Human communication is inherently multimodal. The information conveyed through body language, facial expression, gaze, intonation, speaking style etc. are all important components of everyday communication. An issue within computer science concerns how to provide multimodal agent based systems. Those are systems that interact with users through several channels. These systems can include Virtual Humans. A Virtual Human might for example be a complete creature, i.e. a creature with a whole body including head, arms, legs etc. but it might also be a creature with only a head, a Talking Head. The aim of the Virtual Human Markup Language (VHML) is to control Virtual Humans regarding speech, facial animation, facial gestures and body animation. These parts have previously been implemented and investigated separately, but VHML aims to combine them. In this thesis VHML is verified, validated and evaluated in order to reach that aim and thus VHML is made more solid, homogenous and complete. Further, a Virtual Human has to communicate with the user and even though VHML supports a number of other ways of communication, an important communication channel is speech. The Virtual Human has to be able to interact with the user, therefore a dialogue between the user and the Virtual Human has to be created. These dialogues tend to expand tremendously, hence the Dialogue Management Tool (DMT) was developed. Having a toolmakes it easier for programmers to create and maintain dialogues for the interaction. Finally, in order to demonstrate the work done in this thesis a Talking Head application, The Mystery at West Bay Hospital, has been developed and evaluated. This has shown the usefulness of the DMT when creating dialogues. The work that has been accomplished within this project has contributed to simplify the development of Talking Head applications.

<p>Human communication is inherently multimodal. The information conveyed through body language, facial expression, gaze, intonation, speaking style etc. are all important components of everyday communication. An issue within computer science concerns how to provide multimodal agent based systems. Those are systems that interact with users through several channels. These systems can include Virtual Humans. A Virtual Human might for example be a complete creature, i.e. a creature with a whole body including head, arms, legs etc. but it might also be a creature with only a head, a Talking Head. The aim of the Virtual Human Markup Language (VHML) is to control Virtual Humans regarding speech, facial animation, facial gestures and body animation. These parts have previously been implemented and investigated separately, but VHML aims to combine them. In this thesis VHML is verified, validated and evaluated in order to reach that aim and thus VHML is made more solid, homogenous and complete. Further, a Virtual Human has to communicate with the user and even though VHML supports a number of other ways of communication, an important communication channel is speech. The Virtual Human has to be able to interact with the user, therefore a dialogue between the user and the Virtual Human has to be created. These dialogues tend to expand tremendously, hence the Dialogue Management Tool (DMT) was developed. Having a toolmakes it easier for programmers to create and maintain dialogues for the interaction. Finally, in order to demonstrate the work done in this thesis a Talking Head application, The Mystery at West Bay Hospital, has been developed and evaluated. This has shown the usefulness of the DMT when creating dialogues. The work that has been accomplished within this project has contributed to simplify the development of Talking Head applications.</p>

Baigiamajame darbe tiriamas lietuvių kalbos garsų ir jų vaizdinės informacijos ryššys. Atliekama kalbančių galvų modelių animavimo algoritmų analizė, išškeliama jų problematika ir atsižvelgiant į tai pasiūloma lietuvių kalbos sintetinimo metodika, kuri yra pagrįsta anglų kalbos vizemų naudojimu. ŠŠiame darbe sukuriama 30 trimačių lietuvių kalbos vizemų, kurias vizualiai lyginant su standartinėmis anglų kalbos fonemų vizemomis, sudaroma lietuvišškų fonemų ir anglišškų vizemų atitikčių lentelė. Sudaryta lentelė naudojama lietuvių kalbos garso rinkmenai animuoti. / The connection of Lithuanian sounds and their visual aspect is analyzed. The thesis consists of talking head animation algorithms analysis, problematic topics. In reference it is proposed the idea, how to synthesize Lithuanian speech using English visemes. 30 three dimensional Lithuanian visemes are created. After visual comparison of 3D Lithuanian and standard English visemes, the table of Lithuanian phonemes and English visemes mapping is created. The table is used for animating the Lithuanian sound file.

Is it possible to produce DAISY talking books of acceptable quality without manually adjusting the reading order, inserting page numbers, fine-tuning lexicons, and so on? This question is especially urgent with regard to our open educational resources published on the OpenLearn website. This presentation recounts our experiences of a fully automated production process for DAISY talking books using only open source tools.

DAISY has published a Specification for DAISY Protected Digital Talking Book. This paper discusses why such a specification is useful, not only for rightsholders but also for readers with print disabilities. An implementation of PDTB2 is proposed, called dtbprotect. It makes possible to simply produce an encrypted book from a book in DAISY format. It is currently experimented on the Helene Digital Library for the blind. It will be made available open source as to facilitate its implementation by other digital libraries.

Cette recherche a pour objet l’étude des interactions verbales dans les classes de collège pour les élèves nouvellement arrivés en France. Les élèves qui ne possèdent pas une maîtrise suffisante du français bénéficient à leur arrivée d’heures de français tout en suivant quelques cours dans une classe ordinaire de collège. Mais l’échec scolaire des élèves migrants reste trop fréquent et les chercheurs s’accordent pour dire que l’inadaptation se joue dès le passage en classe ordinaire.Nos observations corroborent cet état de fait. Dans les enregistrements réalisés en français, mathématiques et histoire-géographie, les EANA participent normalement aux échanges en structure d’accueil mais ne prennent jamais la parole sans sollicitation de l’enseignant en classe ordinaire. Les EANA sont mal préparés à la langue de l’école. Que proposent les méthodologies du FLS et du français langue de scolarisation ? Comment former les élèves aux compétences de compréhension et de production de l’oral scolaire compte tenu du contexte particulier de la classe ?Nous avons analysé les interactions verbales d’un corpus de dix-huit séances afin d’observer comment s’organisent les échanges didactiques au collège à la fois en structure d’accueil et en classe ordinaire. Nous cherchons ainsi à déterminer quel contexte interactif peut favoriser l’apparition de cette prise de parole des débutants, afin de proposer quelques pistes didactiques pour préparer les élèves à se rendre acteurs dans la construction interactive des cours auxquels ils participent. / This research deals with the study of verbal interactions in middle school classrooms for pupils who have recently arrived in France. Pupils who do not sufficiently master the French language get French lessons upon their arrival while attending other lessons in a regular middle school classroom. Nevertheless the academic failure of migrant pupils remains too frequent and researchers agree to say that the unsuitability emerges as soon as the pupils integrate regular classes.Our observations support this established fact. On the recordings we have made during French, mathematics, history and geography lessons, the allophone pupils participate normally in the exchanges when they are in the integration structure but never intervene without being invited to by the teacher in the regular classes. The EANA are not well prepared to schooling language. What do the methodologies of the FLS (French as a secondary language) and of the French language of the schooling propose? How can the pupils be trained to oral production and comprehension skills taking into account the particular context of the classroom?We have analysed the verbal interactions of a corpus of eighteen lessons in order to study the organisation of didactic exchanges in middle-schools, both in integration structures and regular classes. We are thus trying to determine which interactive context can help the beginner to start speaking, in order to propose some didactic ideas to prepare pupils to become actors in the interactive construction of the lessons to which they participate in.

Visual speech synthesis deals with synthesising facial animation from an audio representation of speech. In the last decade or so, data-driven approaches have gained prominence with the development of Machine Learning techniques that can learn an audio-visual mapping. Many of these Machine Learning approaches learn a generative model of speech production using the framework of probabilistic graphical models, through which efficient inference algorithms can be developed for synthesis. In this work, the audio and visual parameters are assumed to be generated from an underlying latent space that captures the shared information between the two modalities. These latent points evolve through time according to a dynamical mapping and there are mappings from the latent points to the audio and visual spaces respectively. The mappings are modelled using Gaussian processes, which are non-parametric models that can represent a distribution over non-linear functions. The result is a non-linear state-space model. It turns out that the state-space model is not a very accurate generative model of speech production because it assumes a single dynamical model, whereas it is well known that speech involves multiple dynamics (for e.g. different syllables) that are generally non-linear. In order to cater for this, the state-space model can be augmented with switching states to represent the multiple dynamics, thus giving a switching state-space model. A key problem is how to infer the switching states so as to model the multiple non-linear dynamics of speech, which we address by learning a variable-order Markov model on a discrete representation of audio speech. Various synthesis methods for predicting visual from audio speech are proposed for both the state-space and switching state-space models. Quantitative evaluation, involving the use of error and correlation metrics between ground truth and synthetic features, is used to evaluate our proposed method in comparison to other probabilistic models previously applied to the problem. Furthermore, qualitative evaluation with human participants has been conducted to evaluate the realism, perceptual characteristics and intelligibility of the synthesised animations. The results are encouraging and demonstrate that by having a joint probabilistic model of audio and visual speech that caters for the non-linearities in audio-visual mapping, realistic visual speech can be synthesised from audio speech.

Is it possible to produce DAISY talking books of acceptable quality without manually adjusting the reading order, inserting page numbers, fine-tuning lexicons, and so on? This question is especially urgent with regard to our open educational resources published on the OpenLearn website. This presentation recounts our experiences of a fully automated production process for DAISY talking books using only open source tools.

DAISY book production at the ONCE, with the digitization and subsequent reformatting of our stacks to the DAISY standard, has pursued two key objectives: to digitize our collection as quickly and efficiently as possible and to lower the production costs of digital talking books in DAISY format. This discussion describes the highlights of that process.

No description available.