This thesis presents work in the area of computer-animatedtalking heads. A system for multimodal speech synthesis hasbeen developed, capable of generating audiovisual speechanimations from arbitrary text, using parametrically controlled3D models of the face and head. A speech-specific directparameterisation of the movement of the visible articulators(lips, tongue and jaw) is suggested, along with a flexiblescheme for parameterising facial surface deformations based onwell-defined articulatory targets. To improve the realism and validity of facial and intra-oralspeech movements, measurements from real speakers have beenincorporated from several types of static and dynamic datasources. These include ultrasound measurements of tonguesurface shape, dynamic optical motion tracking of face pointsin 3D, as well as electromagnetic articulography (EMA)providing dynamic tongue movement data in 2D. Ultrasound dataare used to estimate target configurations for a complex tonguemodel for a number of sustained articulations. Simultaneousoptical and electromagnetic measurements are performed and thedata are used to resynthesise facial and intra-oralarticulation in the model. A robust resynthesis procedure,capable of animating facial geometries that differ in shapefrom the measured subject, is described. To drive articulation from symbolic (phonetic) input, forexample in the context of a text-to-speech system, bothrule-based and data-driven articulatory control models havebeen developed. The rule-based model effectively handlesforward and backward coarticulation by targetunder-specification, while the data-driven model uses ANNs toestimate articulatory parameter trajectories, trained ontrajectories resynthesised from optical measurements. Thearticulatory control models are evaluated and compared againstother data-driven models trained on the same data. Experimentswith ANNs for driving the articulation of a talking headdirectly from acoustic speech input are also reported. A flexible strategy for generation of non-verbal facialgestures is presented. It is based on a gesture libraryorganised by communicative function, where each function hasmultiple alternative realisations. The gestures can be used tosignal e.g. turn-taking, back-channelling and prominence whenthe talking head is employed as output channel in a spokendialogue system. A device independent XML-based formalism fornon-verbal and verbal output in multimodal dialogue systems isproposed, and it is described how the output specification isinterpreted in the context of a talking head and converted intofacial animation using the gesture library. Through a series of audiovisual perceptual experiments withnoise-degraded audio, it is demonstrated that the animatedtalking head provides significantly increased intelligibilityover the audio-only case, in some cases not significantly belowthat provided by a natural face. Finally, several projects and applications are presented,where the described talking head technology has beensuccessfully employed. Four different multimodal spokendialogue systems are outlined, and the role of the talkingheads in each of the systems is discussed. A telecommunicationapplication where the talking head functions as an aid forhearing-impaired users is also described, as well as a speechtraining application where talking heads and languagetechnology are used with the purpose of improving speechproduction in profoundly deaf children. / QC 20100506

Der Trend hin zu virtueller Realität (VR) hat neues Interesse an Themen wie der Modellierung menschlicher Körper geweckt, da sich neue Möglichkeiten für Unterhaltung, Konferenzsysteme und immersive Anwendungen bieten. Diese Dissertation stellt deshalb neue Ansätze für die Erstellung animierbarer/realistischer 3D-Kopfmodelle, zur computergestützten Gesichtsanimation aus Text/Sprache sowie zum fotorealistischen Echtzeit-Rendering vor. Um die 3D-Erfassung zu vereinfachen, wird ein hybrider Ansatz genutzt, der statistische Kopfmodelle mit dynamischen Texturen kombiniert. Das Modell erfasst Kopfhaltung und großflächige Deformationen, während die Texturen feine Details und komplexe Bewegungen kodieren. Anhand der erfassten Daten wird ein generatives Modell trainiert, das realistische Gesichtsausdrücke aus einem latenten Merkmalsvektor rekonstruiert. Zudem wird eine neue neuronale Rendering-Technik presentiert, die lernt den Vordergrund (Kopf) vom Hintergrund zu trennen. Das erhöht die Flexibilität während der Inferenz (z. B. neuer Hintergrund) und vereinfacht den Trainingsprozess, da die Segmentierung nicht vorab berechnet werden muss. Ein neuer Animationsansatz ermöglicht die automatische Synthese von Gesichtsvideos auf der Grundlage weniger Trainingssequenzen. Im Gegensatz zu bestehenden Arbeiten lernt das Verfahren einen latenten Merkmalsraum, der sowohl Emotionen als auch visuelle Variationen der Sprache erfasst, während gelernte Priors Animations-Artefakte und unrealistische Kopfbewegungen minimieren. Nach dem Training ist es möglich, realistische Sprachsequenzen zu erzeugen, während der latente Stil-Raum zusätzliche Gestaltungsmöglichkeiten bietet. Die vorgestellten Methoden bilden ein Komplettsystem für die realistische 3D-Modellierung, Animation und Darstellung von menschlichen Köpfen, das den Stand der Technik übertrifft. Dies wird in verschiedenen Experimenten, Ablations-/Nutzerstudien gezeigt und ausführlich diskutiert. / The recent trend of virtual reality (VR) has sparked new interest in human body modeling by offering new possibilities for entertainment, conferencing, and immersive applications (e.g., intelligent virtual assistants). Therefore, this dissertation presents new approaches to creating animatable and realistic 3D head models, animating human faces from text/speech, and the photo-realistic rendering of head models in real-time. To simplify complex 3D face reconstruction, a hybrid approach is introduced that combines a lightweight statistical head model for 3D geometry with dynamic textures. The model captures head orientation and large-scale deformations, while textures encode fine details and complex motions. A deep variational autoencoder trained on these textured meshes learns to synthesize realistic facial expressions from a compact vector. Additionally, a new neural-rendering technique is proposed that separates the head (foreground) from the background, providing more flexibility during inference (e.g., rendering on novel backgrounds) and simplifying the training process as no segmentation masks have to be pre-computed. This dissertation also presents a new neural-network-based approach to synthesizing novel face animations based on emotional speech videos of an actor. Unlike existing works, the proposed model learns a latent animation style space that captures emotions as well as natural variations in visual speech. Additionally, learned animation priors minimize animation artifacts and unrealistic head movements. After training, the animation model offers temporally consistent editing of the animation style according to the users’ needs. Together, the presented methods provide an end-to-end system for realistic 3D modeling, animation, and rendering of human heads. Various experimental results, ablation studies, and user evaluations demonstrate that the proposed approaches outperform the state-of-the-art.

This thesis presents work done during the last ten years on developing five multimodal spoken dialogue systems, and the empirical user studies that have been conducted with them. The dialogue systems have been multimodal, giving information both verbally with animated talking characters and graphically on maps and in text tables. To be able to study a wider rage of user behaviour each new system has been in a new domain and with a new set of interactional abilities. The five system presented in this thesis are: The Waxholm system where users could ask about the boat traffic in the Stockholm archipelago; the Gulan system where people could retrieve information from the Yellow pages of Stockholm; the August system which was a publicly available system where people could get information about the author Strindberg, KTH and Stockholm; the AdAptsystem that allowed users to browse apartments for sale in Stockholm and the Pixie system where users could help ananimated agent to fix things in a visionary apartment publicly available at the Telecom museum in Stockholm. Some of the dialogue systems have been used in controlled experiments in laboratory environments, while others have been placed inpublic environments where members of the general public have interacted with them. All spoken human-computer interactions have been transcribed and analyzed to increase our understanding of how people interact verbally with computers, and to obtain knowledge on how spoken dialogue systems canutilize the regularities found in these interactions. This thesis summarizes the experiences from building these five dialogue systems and presents some of the findings from the analyses of the collected dialogue corpora. / QC 20100611

Es presenta un esquema únic per a la síntesi i anàlisi audiovisual personalitzable realista de seqüències audiovisuals de cares parlants i seqüències visuals de llengua de signes en àmbit domèstic. En el primer cas, amb animació totalment sincronitzada a través d'una font de text o veu; en el segon, utilitzant la tècnica de lletrejar paraules mitjançant la ma. Les seves possibilitats de personalització faciliten la creació de seqüències audiovisuals per part d'usuaris no experts. Les aplicacions possibles d'aquest esquema de síntesis comprenen des de la creació de personatges virtuals realistes per interacció natural o vídeo jocs fins vídeo conferència des de molt baix ample de banda i telefonia visual per a les persones amb problemes d'oïda, passant per oferir ajuda a la pronunciació i la comunicació a aquest mateix col·lectiu. El sistema permet processar seqüències llargues amb un consum de recursos molt reduït, sobre tot, en el referent a l'emmagatzematge, gràcies al desenvolupament d'un nou procediment de càlcul incremental per a la descomposició en valors singulars amb actualització de la informació mitja. Aquest procediment es complementa amb altres tres: el decremental, el de partició i el de composició. / Se presenta un esquema único para la síntesis y análisis audiovisual personalizable realista de secuencias audiovisuales de caras parlantes y secuencias visuales de lengua de signos en entorno doméstico. En el primer caso, con animación totalmente sincronizada a través de una fuente de texto o voz; en el segundo, utilizando la técnica de deletreo de palabras mediante la mano. Sus posibilidades de personalización facilitan la creación de secuencias audiovisuales por parte de usuarios no expertos. Las aplicaciones posibles de este esquema de síntesis comprenden desde la creación de personajes virtuales realistas para interacción natural o vídeo juegos hasta vídeo conferencia de muy bajo ancho de banda y telefonía visual para las personas con problemas de oído, pasando por ofrecer ayuda en la pronunciación y la comunicación a este mismo colectivo. El sistema permite procesar secuencias largas con un consumo de recursos muy reducido gracias al desarrollo de un nuevo procedimiento de cálculo incremental para la descomposición en valores singulares con actualización de la información media. / A shared framework for realistic and personalizable audiovisual synthesis and analysis of audiovisual sequences of talking heads and visual sequences of sign language is presented in a domestic environment. The former has full synchronized animation using a text or auditory source of information; the latter consists in finger spelling. Their personalization capabilities ease the creation of audiovisual sequences by non expert users. The applications range from realistic virtual avatars for natural interaction or videogames to low bandwidth videoconference and visual telephony for the hard of hearing, including help to speech therapists. Long sequences can be processed with reduced resources, specially storing ones. This is allowed thanks to the proposed scheme for the incremental singular value decomposition with mean preservation. This scheme is complemented with another three: the decremental, the split and the composed ones.