I utilized a design science approach to create an automated kiosk that uses embodied intelligent agents to interview individuals and detect changes in arousal, behavior, and cognitive effort by using psychophysiological information systems. This dissertation achieves three primary purposes.First, I describe the creation of this new Information Technology artifact, discuss design choices, and show the completed prototype.Second, related to this new system, I propose a unique class of intelligent agents, which are described as Special Purpose Embodied Conversational Intelligence with Environmental Sensors (SPECIES). I outline a system model that frames the conceptual components of SPECIES agents, provide design principles for developing SPECIES agents, and discuss some of the research implications of the various components in the model.Third, based on the SPECIES paradigm, I present five studies that evaluate different parts of the model. These studies form the foundational research for the development of the automated kiosk. In the first study, participants interacted with an automated interviewing agent via a chat-based modality (108 participants). The study clearly demonstrates the strong, positive correlation of both response time and the number of times a message is edited to deceitful responses. The software developed became the heart of the kiosk. The second study evaluated changing human decision-making by including influence tactics in decision aids (41 participants). This paper-based decision experiment showed that framing decision aids as appeals to individuals' values possibly change individuals' decisions and was the basis for study 4. The third study examined human-computer interaction and how SPECIES agents can change perceptions of information systems by varying appearance and demeanor (88 participants). Instantiations that had the agents embodied as males were perceived as more powerful, while female embodied agents were perceived as more likeable. Similarly, smiling agents were perceived as more likable than neutral demeanor agents. The fourth study assessed how incorporating impression management techniques into embodied conversational agents can influence human perceptions of the system (88 participants). The impression management techniques proved to be very successful in changing user perceptions. Specifically, agents that performed self-promotion were perceived as more powerful, trustworthy and expert. Agents that performed ingratiation were perceived as more attractive. In the fifth study, I used an embodied agent to interview people who had either constructed a fake bomb and packed it into a bag or had only packed clothes into a bag (60 participants). The agent used eye-tracking technology to capture pupil dilation and gaze behavior. When combined with vocal measurements, the kiosk technology was able to achieve over 93% accuracy in one trial.

It is projected that by the year 2033, 41% of the population in the United Kingdom will be aged 60 and over, compared to 17% in 2010. However, as people get older, age-related impairments to working memory and cognition, as well as sensory impairments, mean that older adults struggle to adopt new information technology systems, which have become integral to everyday life in recent years. The research presented here investigates, across a series of three large scale experiments, how Embodied Conversational Agents (ECAs) can be used in virtual world applications, addressing three important domains of application space, to understand how ECAs may be used to engage with older users. The Virtual Banking Experiment investigates whether or not ECA gender or age portrayal has an effect on overall usability of virtual world applications for older users. The results of the experiment show that older users do not exhibit a significant preference for a particular ECA gender, however they do exhibit a preference for interacting with younger ECAs when compared with older ECAs in these applications. Results of the Entertainment Experiment show that scripting the interaction, between older users and ECAs in a virtual world quiz game, significantly improved the usability of these virtual world gaming applications employing ECAs as interlocators. The Social Support Experiment investigated three styles of providing advice to older users as part of a virtual advice bureau service. The results of the experiment show that older users prefer to receive advice from ECAs that has not been presented as having been obtained from real-world experiences.

Embodied Conversational Agents (ECAs) potentially represent a way to deliver services to the public that would previously have require human staff. Making an ECA available online allows out-of-hours access to information and services, as well as allowing users to access the information anywhere there is an internet connection. As handheld devices grow in popularity and become the primary source of internet connection for many users, it is necessary to examine whether an ECA is appropriate for use on a handheld device, and what factors affect its usability. Over the course of four experiments this research examines how using a handheld device is different from using a PC, how an ECA should be presented on a handheld device, how using an ECA service in a public space affects the experience, and how an ECA should interact with users. It was determined that the usability of an ECA service is not affected by the device on which it is experienced, that on smaller screens or in demanding environments the ECA should be emphasised, and that text should be included in an ECA service as long as the ECA remains intermittently visible. It was also found that usability results from the laboratory can be generalised to the real world, that ECA services are appropriate for all ages and genders, that incorporating disclosure elements into an ECA service is a beneficial feature, and that while financial topics are appropriate to be discussed with an ECA, they are best kept to general rather than personal information. The following chapters present the necessary literary background to the field, before covering each experiment individually, and finally presenting detailed conclusions about the usability of ECAs on handheld devices.

Chatbots are programs that mimic human conversation using Artificial Intelligence (AI). Recent advances in natural language pro- cessing pave the way for chatbots to generate more human-like responses. Therefore, chatbots are finding more complex tasks to perform, such as emotional support which requires both understanding emotions and the ability to properly respond to them. This work presents a chatbot capable of identifying the user's personality and creating responses based on that. During this process, emotion detection is being used to detect and react to users' emotions. The chatbot uses a dynamic knowledge graph to save information as the conversation goes on. A user study confirmed that these additions were both noticeable and improved the user's sense that the chatbot was getting to know them as a person. Long-term, we hope this research will help create chatbots that provide emotional support for caregivers who work with people with dementia.

Prevention is a central tenet of modern healthcare. New ways of informing, educating, and allowing patients to keep track of their health are continually developing. Integrating technology that bridges the gap between healthcare providers and their patients is essential in today's digital age. One emerging idea is the use of home healthcare embodied agents. Embodied agents are conservational interfaces that can interact and look like human beings. They can exhibit both verbal and non-verbal language cues and be capable of having a natural conversation. This kind of technology could help both patients and healthcare providers by giving patients a helpful assistant that can educate and take care of particular healthcare needs through conversations with the patient. This could potentially improve patient outcomes, thereby reducing trips to the healthcare provider's office and relieving the burden on physicians and other healthcare providers. This research will focus on home healthcare embodied agents' priorities and the opportunities that this technology can present. A literature review was done to examine the current use of embodied agents in home healthcare settings to establish their present capabilities and applications. Additionally, the strategies/techniques used by home healthcare workers interacting with patients were investigated. By examining the provider-patient relationship and the use of home healthcare embodied agents in tandem, related findings were identified and potential gaps in current research were found that may help direct future exploration.

Meaningful social connections are an important part of our lives, especially as we age, and are associated with life satisfaction and psychological well-being. At the same time making friends and creating connections is known to be challenging in older age. In this thesis, we focus on studying how technology can help to collect information about older adults that can be useful for facilitating friendship formation and social interactions among users. We start by describing early work that shows the opportunities of technology in improving well-being of older adults. The conducted studies and review work highlights the potential of social interactions in motivating older adults for technology use and exercising. We then study factors affecting people's social connectedness and friendships. The study highlights that common life points are related to higher levels of connectedness and frequency of interactions. We then move the focus on studying friendship formation in later life, and specifically on how technology can help to facilitate friendship formation. From observations in the nursing homes we see that reminiscence is often used to collect information about a person’s history and values, we look at this practice as a way to identify information potentially useful to recommend friendships, especially in nursing homes context. We conduct Interviews and observations with nursing homes stakeholders and gerontology doctors to define requirements and opportunities of reminiscence conversational agent suitable to their current practices. We then conduct a study to explore how the concept of the bot and features are perceived by elderly, NH staff and doctors. Finally, we present the work carried out to define and validate the concept of a reminiscence-based conversational agent aimed at: i) conducting storytelling conversations that are engaging and natural and ii) being effective in collecting information about the user (e.g values, interests, places) that later can be used for recommending potential friends.

This thesis presents work in the area of computer-animatedtalking heads. A system for multimodal speech synthesis hasbeen developed, capable of generating audiovisual speechanimations from arbitrary text, using parametrically controlled3D models of the face and head. A speech-specific directparameterisation of the movement of the visible articulators(lips, tongue and jaw) is suggested, along with a flexiblescheme for parameterising facial surface deformations based onwell-defined articulatory targets. To improve the realism and validity of facial and intra-oralspeech movements, measurements from real speakers have beenincorporated from several types of static and dynamic datasources. These include ultrasound measurements of tonguesurface shape, dynamic optical motion tracking of face pointsin 3D, as well as electromagnetic articulography (EMA)providing dynamic tongue movement data in 2D. Ultrasound dataare used to estimate target configurations for a complex tonguemodel for a number of sustained articulations. Simultaneousoptical and electromagnetic measurements are performed and thedata are used to resynthesise facial and intra-oralarticulation in the model. A robust resynthesis procedure,capable of animating facial geometries that differ in shapefrom the measured subject, is described. To drive articulation from symbolic (phonetic) input, forexample in the context of a text-to-speech system, bothrule-based and data-driven articulatory control models havebeen developed. The rule-based model effectively handlesforward and backward coarticulation by targetunder-specification, while the data-driven model uses ANNs toestimate articulatory parameter trajectories, trained ontrajectories resynthesised from optical measurements. Thearticulatory control models are evaluated and compared againstother data-driven models trained on the same data. Experimentswith ANNs for driving the articulation of a talking headdirectly from acoustic speech input are also reported. A flexible strategy for generation of non-verbal facialgestures is presented. It is based on a gesture libraryorganised by communicative function, where each function hasmultiple alternative realisations. The gestures can be used tosignal e.g. turn-taking, back-channelling and prominence whenthe talking head is employed as output channel in a spokendialogue system. A device independent XML-based formalism fornon-verbal and verbal output in multimodal dialogue systems isproposed, and it is described how the output specification isinterpreted in the context of a talking head and converted intofacial animation using the gesture library. Through a series of audiovisual perceptual experiments withnoise-degraded audio, it is demonstrated that the animatedtalking head provides significantly increased intelligibilityover the audio-only case, in some cases not significantly belowthat provided by a natural face. Finally, several projects and applications are presented,where the described talking head technology has beensuccessfully employed. Four different multimodal spokendialogue systems are outlined, and the role of the talkingheads in each of the systems is discussed. A telecommunicationapplication where the talking head functions as an aid forhearing-impaired users is also described, as well as a speechtraining application where talking heads and languagetechnology are used with the purpose of improving speechproduction in profoundly deaf children. / QC 20100506

A great deal of effort has been put into developing Embodied Conversational Agent (ECA) systems that provide a human-like assistant in the user interface. However, little is known whether improvements to ECA interfaces made by such efforts can ever be significant from the users point of view. I studied user experiences with ECA interfaces and evaluated the ECA style of interaction with respect to user expectation, perception, behavior and performance. I introduce a conceptual framework that offers a holistic view of the design space of ECA systems. I also have created a middleware toolkit that facilitates rapid development of application content across different speech and animation platforms. A series of user studies has been carried out to investigate the similarities and differences between human-computer interaction and human-ECA interaction and between human-ECA interaction and human-human interaction. Results from these studies provide strong evidence that people are consciously aware of the capabilities and limitations of ECAs. Traditional GUI design heuristics should be carefully followed when designing ECA interfaces. Furthermore, the results soundly suggest that designers of ECA interfaces take extra care to accommodate individual differences and preferences. Social norms that guide human-human interaction greatly affect individuals expectation and perception of ECA characteristics. The findings support the argument that drawing from both human-computer interaction and human-human interaction can be significantly advantageous to the design of both effective and affective human-ECA interaction.

Embodied Conversational Agents (ECAs) are intelligent software entities with an embodiment used to communicate with users, using natural language. Their purpose is to exhibit the same properties as humans in face-to-face conversation, including the ability to produce and respond to verbal and nonverbal communication. Researchers in the field of ECAs try to create agents that can be more natural, believable and easy to use. Designing an ECA requires understanding that manner, personality, emotion, and appearance are very important issues to be considered. In this thesis, we are interested in increasing believability of ECAs by placing personality at the heart of the human-agent verbal interaction. We propose a model relating personality facets and hidden communication goals that can influence ECA behaviors. Moreover, we apply our model in agents that interact in a puzzle game application. We develop five distinct personality oriented agents using an expressive communication language and a plan-based BDI approach for modeling and managing dialogue according to our proposed model. In summary, we present and test an innovative approach to model mental aspects of ECAs trying to increase their believability and to enhance human-agent affective communication. With this research, we hope to improve the understanding on how ECAs with expressive and affective characteristics can establish and maintain long-term human-agent relationships.

Embodied Conversational Agents (ECAs) are intelligent software entities with an embodiment used to communicate with users, using natural language. Their purpose is to exhibit the same properties as humans in face-to-face conversation, including the ability to produce and respond to verbal and nonverbal communication. Researchers in the field of ECAs try to create agents that can be more natural, believable and easy to use. Designing an ECA requires understanding that manner, personality, emotion, and appearance are very important issues to be considered. In this thesis, we are interested in increasing believability of ECAs by placing personality at the heart of the human-agent verbal interaction. We propose a model relating personality facets and hidden communication goals that can influence ECA behaviors. Moreover, we apply our model in agents that interact in a puzzle game application. We develop five distinct personality oriented agents using an expressive communication language and a plan-based BDI approach for modeling and managing dialogue according to our proposed model. In summary, we present and test an innovative approach to model mental aspects of ECAs trying to increase their believability and to enhance human-agent affective communication. With this research, we hope to improve the understanding on how ECAs with expressive and affective characteristics can establish and maintain long-term human-agent relationships.