I utilized a design science approach to create an automated kiosk that uses embodied intelligent agents to interview individuals and detect changes in arousal, behavior, and cognitive effort by using psychophysiological information systems. This dissertation achieves three primary purposes.First, I describe the creation of this new Information Technology artifact, discuss design choices, and show the completed prototype.Second, related to this new system, I propose a unique class of intelligent agents, which are described as Special Purpose Embodied Conversational Intelligence with Environmental Sensors (SPECIES). I outline a system model that frames the conceptual components of SPECIES agents, provide design principles for developing SPECIES agents, and discuss some of the research implications of the various components in the model.Third, based on the SPECIES paradigm, I present five studies that evaluate different parts of the model. These studies form the foundational research for the development of the automated kiosk. In the first study, participants interacted with an automated interviewing agent via a chat-based modality (108 participants). The study clearly demonstrates the strong, positive correlation of both response time and the number of times a message is edited to deceitful responses. The software developed became the heart of the kiosk. The second study evaluated changing human decision-making by including influence tactics in decision aids (41 participants). This paper-based decision experiment showed that framing decision aids as appeals to individuals' values possibly change individuals' decisions and was the basis for study 4. The third study examined human-computer interaction and how SPECIES agents can change perceptions of information systems by varying appearance and demeanor (88 participants). Instantiations that had the agents embodied as males were perceived as more powerful, while female embodied agents were perceived as more likeable. Similarly, smiling agents were perceived as more likable than neutral demeanor agents. The fourth study assessed how incorporating impression management techniques into embodied conversational agents can influence human perceptions of the system (88 participants). The impression management techniques proved to be very successful in changing user perceptions. Specifically, agents that performed self-promotion were perceived as more powerful, trustworthy and expert. Agents that performed ingratiation were perceived as more attractive. In the fifth study, I used an embodied agent to interview people who had either constructed a fake bomb and packed it into a bag or had only packed clothes into a bag (60 participants). The agent used eye-tracking technology to capture pupil dilation and gaze behavior. When combined with vocal measurements, the kiosk technology was able to achieve over 93% accuracy in one trial.

It is projected that by the year 2033, 41% of the population in the United Kingdom will be aged 60 and over, compared to 17% in 2010. However, as people get older, age-related impairments to working memory and cognition, as well as sensory impairments, mean that older adults struggle to adopt new information technology systems, which have become integral to everyday life in recent years. The research presented here investigates, across a series of three large scale experiments, how Embodied Conversational Agents (ECAs) can be used in virtual world applications, addressing three important domains of application space, to understand how ECAs may be used to engage with older users. The Virtual Banking Experiment investigates whether or not ECA gender or age portrayal has an effect on overall usability of virtual world applications for older users. The results of the experiment show that older users do not exhibit a significant preference for a particular ECA gender, however they do exhibit a preference for interacting with younger ECAs when compared with older ECAs in these applications. Results of the Entertainment Experiment show that scripting the interaction, between older users and ECAs in a virtual world quiz game, significantly improved the usability of these virtual world gaming applications employing ECAs as interlocators. The Social Support Experiment investigated three styles of providing advice to older users as part of a virtual advice bureau service. The results of the experiment show that older users prefer to receive advice from ECAs that has not been presented as having been obtained from real-world experiences.

Embodied Conversational Agents (ECAs) potentially represent a way to deliver services to the public that would previously have require human staff. Making an ECA available online allows out-of-hours access to information and services, as well as allowing users to access the information anywhere there is an internet connection. As handheld devices grow in popularity and become the primary source of internet connection for many users, it is necessary to examine whether an ECA is appropriate for use on a handheld device, and what factors affect its usability. Over the course of four experiments this research examines how using a handheld device is different from using a PC, how an ECA should be presented on a handheld device, how using an ECA service in a public space affects the experience, and how an ECA should interact with users. It was determined that the usability of an ECA service is not affected by the device on which it is experienced, that on smaller screens or in demanding environments the ECA should be emphasised, and that text should be included in an ECA service as long as the ECA remains intermittently visible. It was also found that usability results from the laboratory can be generalised to the real world, that ECA services are appropriate for all ages and genders, that incorporating disclosure elements into an ECA service is a beneficial feature, and that while financial topics are appropriate to be discussed with an ECA, they are best kept to general rather than personal information. The following chapters present the necessary literary background to the field, before covering each experiment individually, and finally presenting detailed conclusions about the usability of ECAs on handheld devices.

This thesis presents work in the area of computer-animatedtalking heads. A system for multimodal speech synthesis hasbeen developed, capable of generating audiovisual speechanimations from arbitrary text, using parametrically controlled3D models of the face and head. A speech-specific directparameterisation of the movement of the visible articulators(lips, tongue and jaw) is suggested, along with a flexiblescheme for parameterising facial surface deformations based onwell-defined articulatory targets. To improve the realism and validity of facial and intra-oralspeech movements, measurements from real speakers have beenincorporated from several types of static and dynamic datasources. These include ultrasound measurements of tonguesurface shape, dynamic optical motion tracking of face pointsin 3D, as well as electromagnetic articulography (EMA)providing dynamic tongue movement data in 2D. Ultrasound dataare used to estimate target configurations for a complex tonguemodel for a number of sustained articulations. Simultaneousoptical and electromagnetic measurements are performed and thedata are used to resynthesise facial and intra-oralarticulation in the model. A robust resynthesis procedure,capable of animating facial geometries that differ in shapefrom the measured subject, is described. To drive articulation from symbolic (phonetic) input, forexample in the context of a text-to-speech system, bothrule-based and data-driven articulatory control models havebeen developed. The rule-based model effectively handlesforward and backward coarticulation by targetunder-specification, while the data-driven model uses ANNs toestimate articulatory parameter trajectories, trained ontrajectories resynthesised from optical measurements. Thearticulatory control models are evaluated and compared againstother data-driven models trained on the same data. Experimentswith ANNs for driving the articulation of a talking headdirectly from acoustic speech input are also reported. A flexible strategy for generation of non-verbal facialgestures is presented. It is based on a gesture libraryorganised by communicative function, where each function hasmultiple alternative realisations. The gestures can be used tosignal e.g. turn-taking, back-channelling and prominence whenthe talking head is employed as output channel in a spokendialogue system. A device independent XML-based formalism fornon-verbal and verbal output in multimodal dialogue systems isproposed, and it is described how the output specification isinterpreted in the context of a talking head and converted intofacial animation using the gesture library. Through a series of audiovisual perceptual experiments withnoise-degraded audio, it is demonstrated that the animatedtalking head provides significantly increased intelligibilityover the audio-only case, in some cases not significantly belowthat provided by a natural face. Finally, several projects and applications are presented,where the described talking head technology has beensuccessfully employed. Four different multimodal spokendialogue systems are outlined, and the role of the talkingheads in each of the systems is discussed. A telecommunicationapplication where the talking head functions as an aid forhearing-impaired users is also described, as well as a speechtraining application where talking heads and languagetechnology are used with the purpose of improving speechproduction in profoundly deaf children. / QC 20100506

A great deal of effort has been put into developing Embodied Conversational Agent (ECA) systems that provide a human-like assistant in the user interface. However, little is known whether improvements to ECA interfaces made by such efforts can ever be significant from the users point of view. I studied user experiences with ECA interfaces and evaluated the ECA style of interaction with respect to user expectation, perception, behavior and performance. I introduce a conceptual framework that offers a holistic view of the design space of ECA systems. I also have created a middleware toolkit that facilitates rapid development of application content across different speech and animation platforms. A series of user studies has been carried out to investigate the similarities and differences between human-computer interaction and human-ECA interaction and between human-ECA interaction and human-human interaction. Results from these studies provide strong evidence that people are consciously aware of the capabilities and limitations of ECAs. Traditional GUI design heuristics should be carefully followed when designing ECA interfaces. Furthermore, the results soundly suggest that designers of ECA interfaces take extra care to accommodate individual differences and preferences. Social norms that guide human-human interaction greatly affect individuals expectation and perception of ECA characteristics. The findings support the argument that drawing from both human-computer interaction and human-human interaction can be significantly advantageous to the design of both effective and affective human-ECA interaction.

Embodied Conversational Agents (ECAs) are intelligent software entities with an embodiment used to communicate with users, using natural language. Their purpose is to exhibit the same properties as humans in face-to-face conversation, including the ability to produce and respond to verbal and nonverbal communication. Researchers in the field of ECAs try to create agents that can be more natural, believable and easy to use. Designing an ECA requires understanding that manner, personality, emotion, and appearance are very important issues to be considered. In this thesis, we are interested in increasing believability of ECAs by placing personality at the heart of the human-agent verbal interaction. We propose a model relating personality facets and hidden communication goals that can influence ECA behaviors. Moreover, we apply our model in agents that interact in a puzzle game application. We develop five distinct personality oriented agents using an expressive communication language and a plan-based BDI approach for modeling and managing dialogue according to our proposed model. In summary, we present and test an innovative approach to model mental aspects of ECAs trying to increase their believability and to enhance human-agent affective communication. With this research, we hope to improve the understanding on how ECAs with expressive and affective characteristics can establish and maintain long-term human-agent relationships.

Embodied Conversational Agents (ECAs) are intelligent software entities with an embodiment used to communicate with users, using natural language. Their purpose is to exhibit the same properties as humans in face-to-face conversation, including the ability to produce and respond to verbal and nonverbal communication. Researchers in the field of ECAs try to create agents that can be more natural, believable and easy to use. Designing an ECA requires understanding that manner, personality, emotion, and appearance are very important issues to be considered. In this thesis, we are interested in increasing believability of ECAs by placing personality at the heart of the human-agent verbal interaction. We propose a model relating personality facets and hidden communication goals that can influence ECA behaviors. Moreover, we apply our model in agents that interact in a puzzle game application. We develop five distinct personality oriented agents using an expressive communication language and a plan-based BDI approach for modeling and managing dialogue according to our proposed model. In summary, we present and test an innovative approach to model mental aspects of ECAs trying to increase their believability and to enhance human-agent affective communication. With this research, we hope to improve the understanding on how ECAs with expressive and affective characteristics can establish and maintain long-term human-agent relationships.

Embodied Conversational Agents (ECAs) are intelligent software entities with an embodiment used to communicate with users, using natural language. Their purpose is to exhibit the same properties as humans in face-to-face conversation, including the ability to produce and respond to verbal and nonverbal communication. Researchers in the field of ECAs try to create agents that can be more natural, believable and easy to use. Designing an ECA requires understanding that manner, personality, emotion, and appearance are very important issues to be considered. In this thesis, we are interested in increasing believability of ECAs by placing personality at the heart of the human-agent verbal interaction. We propose a model relating personality facets and hidden communication goals that can influence ECA behaviors. Moreover, we apply our model in agents that interact in a puzzle game application. We develop five distinct personality oriented agents using an expressive communication language and a plan-based BDI approach for modeling and managing dialogue according to our proposed model. In summary, we present and test an innovative approach to model mental aspects of ECAs trying to increase their believability and to enhance human-agent affective communication. With this research, we hope to improve the understanding on how ECAs with expressive and affective characteristics can establish and maintain long-term human-agent relationships.

This thesis presents a framework for computer assisted language learning using a virtual language teacher. It is an attempt at creating, not only a new type of language learning software, but also a server-based application that collects large amounts of speech material for future research purposes.The motivation for the framework is to create a research platform for computer assisted language learning, and computer assisted pronunciation training.Within the thesis, different feedback strategies and pronunciation error detectors are exploredThis is a broad, interdisciplinary approach, combining research from a number of scientific disciplines, such as speech-technology, game studies, cognitive science, phonetics, phonology, and second-language acquisition and teaching methodologies.The thesis discusses the paradigm both from a top-down point of view, where a number of functionally separate but interacting units are presented as part of a proposed architecture, and bottom-up by demonstrating and testing an implementation of the framework. / QC 20110511

Research endeavors on spoken dialogue systems in the 1990s and 2000s have led to the deployment of commercial spoken dialogue systems (SDS) in microdomains such as customer service automation, reservation/booking and question answering systems. Recent research in SDS has been focused on the development of applications in different domains (e.g. virtual counseling, personal coaches, social companions) which requires more sophistication than the previous generation of commercial SDS. The focus of this research project is the delivery of behavior change interventions based on the brief intervention counseling style via spoken dialogue systems. Brief interventions (BI) are evidence-based, short, well structured, one-on-one counseling sessions. Many challenges are involved in delivering BIs to people in need, such as finding the time to administer them in busy doctors' offices, obtaining the extra training that helps staff become comfortable providing these interventions, and managing the cost of delivering the interventions. Fortunately, recent developments in spoken dialogue systems make the development of systems that can deliver brief interventions possible. The overall objective of this research is to develop a data-driven, adaptable dialogue system for brief interventions for problematic drinking behavior, based on reinforcement learning methods. The implications of this research project includes, but are not limited to, assessing the feasibility of delivering structured brief health interventions with a data-driven spoken dialogue system. Furthermore, while the experimental system focuses on harmful alcohol drinking as a target behavior in this project, the produced knowledge and experience may also lead to implementation of similarly structured health interventions and assessments other than the alcohol domain (e.g. obesity, drug use, lack of exercise), using statistical machine learning approaches. In addition to designing a dialog system, the semantic and emotional meanings of user utterances have high impact on interaction. To perform domain specific reasoning and recognize concepts in user utterances, a named-entity recognizer and an ontology are designed and evaluated. To understand affective information conveyed through text, lexicons and sentiment analysis module are developed and tested.