The implementation of chatbots and service services is becoming more common. The reason for this is that they are constantly available to answer questions, no matter what time it is. To make one chatbot is not just to write questions and specific answers. Communicating with one chatbot should, as much as possible, look like communicating with another human. The purpose of this report is to create a chatbot that will be used at Luleå university of technology and that will answer questions about system science. Furthermore, the goal is to investigate which design principles should be implemented, their possible concretization, when making chatbots, and possibly coming to new principles. The method used to make the chatbot is Design Science Research Methodology (DSRM). DSRM focuses on solving the problem by creating an IT artifact, which in this case is a chatbot. The result of this work is a created chatbot and design principles that were implemented during the development process.

Contributors from the mid 20th century up to now have developed and refined virtual assistants, taking the technology from a set of rules to assistants driven by Artificial Intelligence. Today, virtual assistants can provide value in organisation and support a sustainable society by conducting basic and repetitive tasks, and help reduce inequalities caused by biased advisors on sensitive topics. Despite its prosperity, current research somewhat lack focus on the evaluation of virtual assistants in industrial applications. The purpose of this paper is to evaluate virtual assistants from a technical, economical and organisational perspective, in order to understand their performance and value in an industrial environment. This has been done in collaboration with IBM and a client company which prefers to remain anonymous in this report. In this company, two IBM Watson Assistants are under development; one for the IT Service Desk, and one for the Ethics & Compliance department. To cover all aspects of the virtual assistants’ performance, quantitative and qualitative methods were used by conducting user testings and surveys. In this process, discussions have been conducted with IBM experts and employees of the firm for which the practical implementation has been studied, to gain a general and specific understanding from different perspectives. From this paper, the following can be concluded. First, technological performance can be described using quantitative metrics such as coverage, confidence, precision and helpfulness, and should be complemented using qualitative measures such as user satisfaction and perceived user understanding. Second, specific technological performance is relative and the technical limitations as well as it’s maturity should be used as a complement to the evaluation of the assistants. Third, identified organisational benefits include: • reduced time-to-resolution, • reduced handling time, • all-hour-support, • scalability and • user understanding Conclusions specific for the use cases show that an assistant implemented in a narrower use case, that is the Ethics & Compliance assistant, easier can be implemented and performs relatively well also in less developed environments. A broader use case, such as the IT assistant, requires more effort to perform at a high level but may be even more beneficial than in the narrow use case once sufficiently refined. / Från mitten av 1900-talet har virtuella assistenter utvecklats och förfinats där teknologin gått från en mängd regler till assistenter drivna av artificiell intelligens. Idag kan virtuella assistenter tillföra värde till organisationer och bidra till ett hållbart samhälle bland annat genom att utföra enkla och återkommande uppgifter samt minska ojämlikheter orsakad av partiska rådgivare i känsliga frågor. Trots framgången har nuvarande forskning inte fokuserat på evalueringen av virtuella assistenter i industriella sammanhang. Syftet med denna rapport är att utvärdera virtuella assistenter från ett tekniskt, ekonomiskt och organisationellt perspektiv för att förstå dess prestation i industriella miljöer. Arbetet har genomförts i samarbete med IBM och en av deras kunder som föredrar att förbli anonyma. I detta företag är två IBM Watson Assistant under utveckling; en för deras IT Service Desk och en för deras avdelning för Ethics & Compliance. I studien har både kvantitativa och kvalitativa metoder använts, däribland användartestning och frågeformulär, för att inkludera alla aspekter av de virtuella assistenternas prestation. I denna process har diskussioner förts med experter inom IBM samt medarbetare på företaget för vilket den praktiska implementationen studerats för att få en förståelse för både generell och specifik kunskap ur olika perspektiv. I denna rapport kan följande slutsatser dras. Ett, den tekniska prestationen kan bestämmas med kvantitativa mätetal så som täckning (coverage), säkerhet (confidence), precision och hjälpsamhet (helpfulness), och kompletteras med kvalitativa mätetal som användarnöjdhet och upplevd förståelse för användaren. Två, specifik teknisk prestation är relativ och de tekniska  begränsningarna samt mognad bör användas som komplement till utvärderingen av assistenterna. Tre, identifierade organisationsfördelar inkluderar: • reducerad time-to-resolution, • reducerad hanteringstid, • support ¨öppen dygnet runt, • skalbarhet, och • användarförståelse Slutsatserna i de specifika fallen visar att en virtuell assistent som implementeras inom ett smalare område, som en assistent för Ethics & compliance, enklare kan implementeras samt presterar relativt bra även i en mindre utvecklad miljö. Bredare områden, som en assistent för IT-support, kräver mer arbete för att prestera på en hög nivå men kan vara ännu mer värdefull än assistenten i det smala området när den blivit tillräckligt utvecklad.

Chat-bots are computer programs designed to conduct textual or audible conversations with a single user. The job of a chat-bot is to be able to find the best response for any request the user issues. The best response is considered to answer the question and contain relevant information while following grammatical and lexical rules. Modern chat-bots often have trouble accomplishing all these tasks. State-of-the-art approaches, such as deep learning, and large datasets help chat-bots tackle this problem better. While there is a number of different approaches that can be applied for different kind of bots, datasets of suitable size are not always available. In this work, we introduce and evaluate a method of expanding the size of datasets. This will allow chat-bots, in combination with a good learning algorithm, to achieve higher precision while handling their tasks. The expansion method uses the continual learning approach that allows the bot to expand its own dataset while holding conversations with its users. In this work we test continual learning with IBM Watson Assistant chat-bot as well as a custom case study chat-bot implementation. We conduct the testing using a smaller and a larger datasets to find out if continual learning stays effective as the dataset size increases. The results show that the more conversations the chat-bot holds, the better it gets at guessing the intent of the user. They also show that continual learning works well for larger and smaller datasets, but the effect depends on the specifics of the chat-bot implementation. While continual learning makes good results better, it also turns bad results into worse ones, thus the chat-bot should be manually calibrated should the precision of the original results, measured before the expansion, decrease.