Assessing a conversational agent’s understanding capabilities is critical, as poor user interactions could seal the agent’s fate at the very beginning of its lifecycle with users abandoning the system. In this thesis we explore the use of paraphrases as a testing tool for conversational agents. Paraphrases, which are different ways of expressing the same intent, are generated based on known working input by performing lexical substitutions and by introducing multiple spelling divergences. As the expected outcome for this newly generated data is known, we can use it to assess the agent’s robustness to language variation and detect potential understanding weaknesses. As demonstrated by a case study, we obtain encouraging results as it appears that this approach can help anticipate potential understanding shortcomings, and that these shortcomings can be addressed by the generated paraphrases. / Att bedöma en konversationsagents språkförståelse är kritiskt, eftersom dåliga användarinteraktioner kan avgöra om agenten blir en framgång eller ett misslyckande redan i början av livscykeln. I denna rapport undersöker vi användningen av parafraser som ett testverktyg för dessa konversationsagenter. Parafraser, vilka är olika sätt att uttrycka samma avsikt, skapas baserat på känd indata genom att utföra lexiska substitutioner och genom att introducera flera stavningsavvikelser. Eftersom det förväntade resultatet för denna indata är känd kan vi använda resultaten för att bedöma agentens robusthet mot språkvariation och upptäcka potentiella förståelssvagheter. Som framgår av en fallstudie får vi uppmuntrande resultat, eftersom detta tillvägagångssätt verkar kunna bidra till att förutse eventuella brister i förståelsen, och dessa brister kan hanteras av de genererade parafraserna.

Task-oriented dialogue (ToD) systems need to interpret the user's input to understand the user's needs (intent) and corresponding relevant information (slots). This process is performed by a Natural Language Understanding (NLU) component, which maps the text utterance into a semantic frame representation, involving two subtasks: intent classification (text classification) and slot filling (sequence tagging). Typically, new domains and languages are regularly added to the system to support more functionalities. Collecting domain-specific data and performing fine-grained annotation of large amounts of data every time a new domain and language is introduced can be expensive. Thus, developing an NLU model that generalizes well across domains and languages with less labeled data (low-resource) is crucial and remains challenging. This thesis focuses on investigating transfer learning and data augmentation methods for low-resource NLU in ToD. Our first contribution is a study of the potential of non-conversational text as a source for transfer. Most transfer learning approaches assume labeled conversational data as the source task and adapt the NLU model to the target task. We show that leveraging similar tasks from non-conversational text improves performance on target slot filling tasks through multi-task learning in low-resource settings. Second, we propose a set of lightweight augmentation methods that apply data transformation on token and sentence levels through slot value substitution and syntactic manipulation. Despite its simplicity, the performance is comparable to deep learning-based augmentation models, and it is effective on six languages on NLU tasks. Third, we investigate the effectiveness of domain adaptive pre-training for zero-shot cross-lingual NLU. In terms of overall performance, continued pre-training in English is effective across languages. This result indicates that the domain knowledge learned in English is transferable to other languages. In addition to that, domain similarity is essential. We show that intermediate pre-training data that is more similar – in terms of data distribution – to the target dataset yields better performance.

京都大学 / 新制・課程博士 / 博士(情報学) / 甲第24929号 / 情博第840号 / 新制||情||140(附属図書館) / 京都大学大学院情報学研究科知能情報学専攻 / (主査)特定教授 黒橋 禎夫, 教授 河原 達也, 教授 西野 恒 / 学位規則第4条第1項該当 / Doctor of Informatics / Kyoto University / DFAM

No description available.

What do you think of when you hear the word chatbot? A helpful assistant when booking flight tickets? Maybe a frustrating encounter with a company’s customer support, or smart technologies that will eventually take over your job? The field of chatbots is under constant development and bots are more and more taking a place in our everyday life, but how well do they really understand us humans?  The objective of this thesis is to investigate how capable two open source chatbots are in understanding human language when given input containing spelling errors, synonyms or faulty syntax. The study will further investigate if the bots get better at identifying what the user’s intention is when supplied with more training data to base their analysis on.  Two different chatbot frameworks, Botpress and Rasa, were consulted to execute this experiment. The two bots were created with basic configurations and trained using the same data. The chatbots underwent three rounds of training and testing, where they were given additional training and asked control questions to see if they managed to interpret the correct intent. All tests were documented and scores were calculated to create comparable data. The results from these tests showed that both chatbots performed well when it came to simpler spelling errors and syntax variations. Their understanding of more complex spelling errors were lower in the first testing phase but increased with more training data. Synonyms followed a similar pattern, but showed a minor tendency towards becoming overconfident and producing incorrect results with a high confidence in the last phase. The scores pointed to both chatbots getting better at understanding the input when receiving additional training. In conclusion, both chatbots showed signs of understanding language variations when given minimal training, but got significantly better results when provided with more data. The potential to create a bot with a substantial understanding of human language is evident with these results, even for developers who are previously not experienced with creating chatbots, also taking into consideration the vast possibilities to customise your chatbot.

No description available.

Spoken dialogue systems are application interfaces that enable humans to interact with computers using spoken natural language. A major challenge for these systems is dealing with the ubiquity of variability—in user behavior, in the performance of the various speech and language processing sub-components, and in the dynamics of the task domain. However, as the predominant methodology for dialogue system development is to handcraft the sub-components, these systems typically lack robustness in user interactions. Data-driven methods, on the other hand, have been shown to offer robustness to variability in various domains of computer science and are increasingly being used in dialogue systems research.     This thesis makes four novel contributions to the data-driven methods for spoken dialogue system development. First, a method for interpreting the meaning contained in spoken utterances is presented. Second, an approach for determining when in a user’s speech it is appropriate for the system to give a response is presented. Third, an approach for error detection and analysis in dialogue system interactions is reported. Finally, an implicitly supervised learning approach for knowledge acquisition through the interactive setting of spoken dialogue is presented.      The general approach taken in this thesis is to model dialogue system tasks as a classification problem and investigate features (e.g., lexical, syntactic, semantic, prosodic, and contextual) to train various classifiers on interaction data. The central hypothesis of this thesis is that the models for the aforementioned dialogue system tasks trained using the features proposed here perform better than their corresponding baseline models. The empirical validity of this claim has been assessed through both quantitative and qualitative evaluations, using both objective and subjective measures. / Den här avhandlingen utforskar datadrivna metoder för utveckling av talande dialogsystem. Motivet bakom sådana metoder är att dialogsystem måste kunna hantera en stor variation, i såväl användarnas beteende, som i prestandan hos olika tal- och språkteknologiska delkomponenter. Traditionella tillvägagångssätt, som baseras på handskrivna komponenter i dialogsystem, har ofta svårt att uppvisa robusthet i hanteringen av sådan variation. Datadrivna metoder har visat sig vara robusta mot variation i olika problem inom datavetenskap och artificiell intelligens, och har på senare tid blivit populära även inom forskning kring talande dialogsystem. Den här avhandlingen presenterar fyra nya bidrag till datadrivna metoder för utveckling av talande dialogsystem. Det första bidraget är en datadriven metod för semantisk tolkning av talspråk. Den föreslagna metoden har två viktiga egenskaper: robust hantering av ”ogrammatisk” indata (på grund av talets spontana natur samt fel i taligenkänning), samt bevarande av strukturella relationer mellan koncept i den semantiska representationen. Tidigare metoder för semantisk tolkning av talspråk har typiskt sett endast hanterat den ena av dessa två utmaningar. Det andra bidraget i avhandlingen är en datadriven metod för turtagning i dialogsystem. Den föreslagna modellen utnyttjar prosodi, syntax, semantik samt dialogkontext för att avgöra när i användarens tal som det är lämpligt för systemet att ge respons. Det tredje bidraget är en data-driven metod för detektering av fel och missförstånd i dialogsystem. Där tidigare arbeten har fokuserat på detektering av fel on-line och endast testats i enskilda domäner, presenterats här modeller för analys av fel såväl off-line som on-line, och som tränats samt utvärderats på tre skilda dialogsystemkorpusar. Slutligen presenteras en metod för hur dialogsystem ska kunna tillägna sig ny kunskap genom interaktion med användaren. Metoden är utvärderad i ett scenario där systemet ska bygga upp en kunskapsbas i en geografisk domän genom så kallad "crowdsourcing". Systemet börjar med minimal kunskap och använder den talade dialogen för att både samla ny information och verifiera den kunskap som inhämtats. Den generella ansatsen i den här avhandlingen är att modellera olika uppgifter för dialogsystem som  klassificeringsproblem, och undersöka särdrag i diskursens kontext som kan användas för att träna klassificerare. Under arbetets gång har olika slags lexikala, syntaktiska, prosodiska samt kontextuella särdrag undersökts. En allmän diskussion om dessa särdrags bidrag till modellering av ovannämnda uppgifter utgör ett av avhandlingens huvudsakliga bidrag. En annan central del i avhandlingen är att träna modeller som kan användas direkt i dialogsystem, varför endast automatiskt extraherbara särdrag (som inte kräver manuell uppmärkning) används för att träna modellerna. Vidare utvärderas modellernas prestanda på såväl taligenkänningsresultat som transkriptioner för att undersöka hur robusta de föreslagna metoderna är. Den centrala hypotesen i denna avhandling är att modeller som tränas med de föreslagna kontextuella särdragen presterar bättre än en referensmodell. Giltigheten hos denna hypotes har bedömts med såväl kvalitativa som kvantitativa utvärderingar, som nyttjar både objektiva och subjektiva mått. / <p>QC 20160225</p>

There is an overwhelming variation in the ways an intelligent agent can rationalize communication with a conversational partner. This variation presents many incompatibilities that lead to the specialization of conversational capabilities. This has produced a plethora of models and ideas on how an intelligent agent should understand, interact with, and incorporate communication from a human conversational participant. This dissertation approaches this problem with the thesis that there exists a language between that of human natural language and the behavioral reasoning of an intelligent agent, and that this language is capable of not only unifying the various models used in literature, but also provides the foundation for a theoretical framework for an engineering methodology for building such models. A theory of practical communication language is developed, including the introduction of the meaning-action concept, an expressive and powerful representation based on speech-act and dialogue-act theories, but extended with notions of behavioral operators as well as signatures that allow the operators to incorporate structured and well-defined concepts. An engineering methodology is presented for the construction of concepts, operators and rules that create the language and model of a specific domain, including methodology for the verification and validation of that language and model. The resultant practical communication language methodology, based on the combination of rational communication and meaning-action concepts, will introduce several major enhancements to dialogue management. These enhancements include the use of meaning-action concepts as a shared medium and the introduction of a shared concept graph. This methodology will be used along with various dialogue models from human-human, human-agent and agent-agent communication to construct a task-oriented language and model called the task communication language framework. This framework is then implemented within an intelligent agent in a real-time resource management simulation. A sample output listing from actual human interaction with that implementation is used to demonstrate that the resulting framework does indeed incorporate many of the disparate models of communication and their corresponding capabilities including command and control, information seeking, notification and bother, clarification, explanation, discussion, negotiation, mutual planning, interruption, feedback, adjustable autonomy and corrective dialogues.

Mentaliseringsförmåga innebär förmågan att ta en annan persons perspektiv, att förstå hur någon annan tänker och känner. Det innebär även att förstå de egna tankarna och reaktionerna relaterat till andra personers tankar och känslor. Det är viktigt med en välfungerande mentaliseringsförmåga för att kunna samverka med andra individer och sin omgivning på ett pragmatiskt och ändamålsenligt sätt. Det finns flera olika förmågor som kan vara viktiga för mentaliseringsförmågan, i vilken grad de påverkar är dock fortfarande oklart. Syftet med detta arbete var att undersöka mentaliseringsförmågan och dess samvariation med andra kognitiva förmågor hos barn i åldrarna 6-7 år. I föreliggande studie användes tio test för att undersöka vilka kognitiva förmågor som samverkade med mentaliseringsförmågan. De förmågor som testades var visuellt och auditivt arbetsminne, korttidsminne, språkförståelse och ickeverbal intelligens. Testgruppen bestod av 25 typiskt utvecklade barn i åldrarna 6:0–8:0 år med svenska som modersmål. Resultatet av testerna visade att ickeverbal intelligens, korttidsminne och språkförståelse korrelerade med barnens mentaliseringsförmåga. Vad gällde arbetsminnet verkade det främst som att en arbetsminneskapacitet upp till en viss nivå gynnade mentaliseringsförmågan, kapacitet över denna nivå verkade inte ha någon betydelse för prestationen. Det upptäcktes ingen enskild faktor som var viktigare än de andra för mentaliseringsförmågan, utan att samverkan av dessa förmågor är viktig. / Theory of mind, or mentalization ability, is the ability to understand how another individual thinks, acts and feels. It is important to develop a mentalization ability in order to interact with other people and the surrounding social environment in a pragmatic way. There are several abilities that might play an important role in the developmental process of Theory of mind. It is still uncertain to which degree these abilities effect the mentalization ability. The purpose with this study was to investigate the mentalization ability and its relationship with other cognitive abilities in children aged 6-7 years. In this study, ten different tests were used to analyze which abilities correlated with Theory of mind. The abilities that were tested were visual- and auditory working memory, short-term memory, non-verbal intelligence and language understanding. The participating test group consisted of 25 typically developed children aged 6:0-8:0 with Swedish as mother tongue. The results from the tests showed that the mentalization ability correlated with nonverbal intelligence, short-term memory and language understanding. It appears that a certain level of working memory is important, but that an exceptionally good working memory will not improve the mentalization ability further. The result showed that no single ability were more important than the others for the mentalization ability.

Generating computer animation from natural language instructions is a complex task that encompasses several key aspects of artificial intelligence including natural language understanding, computer graphics and knowledge representation. Traditionally, this task has been approached using rule based systems which were highly successful on their respective domains, but were difficult to generalise to other domains. In this thesis, I describe the key theories and principles behind a domain-independent machine learning framework for constructing natural language based animation systems, and show how this framework can be more flexible and more powerful than the prevalent rule based approach. / I begin this thesis with a thorough introduction to the goals of the research. I then review the most relevant literature to put this research into perspective. After the literature review, I provide brief descriptions to the most relevant technologies in both natural language processing and computer graphics. I then report original research in semantic role labelling and verb sense disambiguation, followed by a detailed description and analysis of the machine learning framework for natural language based animation generation. / The key contributions of this thesis are: a novel method for performing semantic role labelling of prepositional phrases, a novel method for performing verb sense disambiguation, and a novel machine learning framework for grounding linguistic information in virtual worlds and converting verb-semantic information to computer graphics commands to create computer animation.