This thesis investigates the implementation of an Retrieval-Augmented Generation (RAG) Teamschat bot to enhance the efficiency of a service organization, utilizing Microsoft Azure’s AI services.The project combines the retrieval capabilities of Azure AI Search with OpenAI’s GPT-3.5 Turboand Meta’s Llama 3 70B-instruct. The aim is to develop a chat bot capable of handling bothstructured and unstructured data. The motivation for this work comes from the limitations ofstandalone Large Language Models (LLMs) which often fail to provide accurate and contextuallyrelevant answers without external knowledge. The project uses the retriever and two languagemodels and evaluates them using F1 scoring. The retriever performs well, but the RAG modelproduces wrong or too long answers. Metrics other than F1 scoring could be used, and future workin prompt engineering as well as larger test datasets could improve model performance.

The Thermo-Calc software is a key tool in the research process for many material engineers. However, integrating multiple modules in Thermo-Calc requires the user to write code in a Python-based language, which can be challenging for novice programmers. This project aims to enable the generation of such code from user prompts by using existing generative AI models. In particular, we use a retrieval-augmented generation architecture applied to LLaMA and Mistral models. We use Code LLaMA-Instruct models with 7, 13, and 34 billion parameters, and a Mistral-Instruct model with 7 billion parameters. These models are all based on LLaMA 2. We also use a LLaMA 3-Instruct model with 8 billion parameters. All these models are instruction-tuned, which suggests that they have the capability to interpret natural language and identify appropriate options for a command-line program such as Python. In our testing, the LLaMA 3-Instruct model performed best, achieving 53% on the industry benchmark HumanEval and 49% on our internal adequacy assessment at pass@1, which is the expected probability of getting a correct solution when generating a response. This indicates that the model generates approximately every other answer correct. Due to GPU memory limitations, we had to apply quantisation to process the 13 and 34 billion parameter models. Our results revealed a mismatch between model size and optimal levels of quantisation, indicating that reduced precision adversely affects the performance of these models. Our findings suggest that a properly customised large language model can greatly reduce the coding effort of novice programmers, thereby improving productivity in material research.

Large Language Models (LLMs) have emerged as a revolutionary force in Computer Science Education, offering unprecedented opportunities to facilitate learning and comprehension. Their application in the classroom, however, is not without challenges. LLMs are prone to hallucination and contextual inaccuracies. Furthermore, they risk exposing learning processes to cheating illicit practices and providing explicit solutions that impede the development of critical thinking skills in students. To address these pitfalls and investigate how specialized LLMs can enhance engagement among learners particularly using LLMs, we present Gurukul, a unique coding platform incorporating dual features - Retrieval Augmented Generation and Guardrails. Gurukul's practice feature provides a hands-on code editor to solve DSA problems with the help of a dynamically Guardrailed LLM to prevent explicit code solutions. On the other hand, Gurukul's Study feature incorporates a Retrieval Augmented Generation mechanism that uses OpenDSA as its source of truth, allowing the LLM to fetch and present information accurately and relevantly, thereby trying to overcome the issue of inaccuracies. We present these features to evaluate the user perceptions of LLM-assisted educational tools. To evaluate the effectiveness and utility of Gurukul in a real-world educational setting, we conducted a User Study and a User Expert Review with students (n=40) and faculty (n=2), respectively, from a public state university in the US specializing in DSA courses. We examine student's usage patterns and perceptions of the tool and report reflections from instructors and a series of recommendations for classroom use. Our findings suggest that Gurukul had a positive impact on student learning and engagement in learning DSA. This feedback analyzed through qualitative and quantitative methods indicates the promise of the utility of specialized LLMs in enhancing student engagement in DSA learning. / Master of Science / Computer science education is continuously evolving with new technologies enhancing the learning experience. This thesis introduces Gurukul, an innovative platform designed to transform the way students learn Data Structures and Algorithms (DSA). Gurukul integrates large language models (LLMs) with advanced features like Retrieval Augmented Generation (RAG) and Guardrails to create an interactive and adaptive learning environment. Traditional learning methods often struggle with providing accurate information and engaging students actively. Gurukul addresses these issues by offering a live code editor for hands-on practice and a study feature that retrieves accurate information from trusted sources. The platform ensures students receive context-sensitive guidance without bypassing critical thinking skills. A study involving students and faculty from a public university specializing in DSA courses evaluated Gurukul's effectiveness. The feedback, based on qualitative and quantitative evaluations, highlights the platform's potential to enhance student engagement and learning outcomes in computer science education. This research contributes to the ongoing development of educational technologies and provides insights for future improvements.

Vikten av hållbarhetsredovisning kan ses genom den uppmärksamhet ämnet har från företag, media, myndigheter och den ökande regleringen genom införandet av nya direktiv och lagstiftning. Att manuellt analysera företags hållbarhetsredovisningar är en tidskrävande process. En automatiserad analys av hållbarhetsredovisningar skulle innebära ekonomiska och tidsmässiga vinster när viktiga insikter tas fram relaterat till större företags påverkan på sin miljö och omgivning. Denna studie syftar till att utforska möjligheterna till en automatisering av en befintlig manuell arbetsmetod. Prototypen som utvecklats tillämpar moderna språkbehandlingsmetoder, ett område inom maskininlärning, för att realisera denna vision. Studiens implementation uppnår för de utvärderade språkmodellerna upp till 96% precision för majoritetsklassen vid bearbetning av grunddatat respektive 55% precision för minoritetsdataklassen vid bearbetning av grunddata jämfört resultat från den manuellt genomförda metoden. Slutsatsen är att en automatiserad version av den befintliga manuella analysmetoden kan konstrueras och även förbättras med den snabba utveckling som sker inom teknologi och språkmodeller, om ytterligare resurser avsätts. Resultaten visar hopp om potentialen för en metodik som utvecklas i vidare arbeten. / The importance of sustainability reporting can be observed by the attention directed towards the subject from companies, media and authorities’ continuous new directives and laws. To manually analyze companies’ sustainability reports is a time-consuming process. An automated approach analyzing sustainability reports would give advantages regarding both time and economics when important insights related to companies’ operations are brought into light. This study aims to explore possibilities in automating an existing manual method related to analyzing sustainability reports. The developed prototype applies modern language models and methods related to machine learning to realize this vision. For the evaluated language models, the study’s implementation achieves up to 96% precision for the majority class, while the minority class achieves up to 55% precision in processing of data, when compared to reference results from the manual evaluation method. The work’s conclusion indicates that an automated version of the existing manual method for analysis can be constructed with sufficient resources, and even further improved as the area of technology further advances. The results are positive for the potential for a more sophisticated method that can be developed in further work.

This thesis presents the design and implementation of a generative Artificial Intelligence (AI)-based decision-support interface applied to the domain of pub- lic transport leveraging both offline and logged data from both past records and real-time updates. The AI assistant system was developed leveraging pre- trained Large Language Models (LLMs) together with Retrieval Augmented Generation (RAG) and the Function Calling Application Programming Inter- face (API), provided by OpenAI, for automating the process of adding knowl- edge to the LLM. Challenges such as formatting and restructuring of data, data retrieval methodologies, accuracy and latency were considered. The result is an AI assistant which can have a conversation with users, answer questions re- garding departures, arrivals, specific vehicle trips, and other questions relevant within the domain of the dataset. The AI assistant system has also been devel- oped to provide client-side actions that integrate with the user interface, enabling interactive elements such as clickable links to trigger relevant actions based on the content provided Different LLMs, including GPT-3.5 and GPT-4 with different temperatures, were compared and evaluated with a pre-defined set of questions paired with a respective ground truth. By adopting a conversational approach, the project aims to streamline infor- mation extraction from extensive datasets, offering a more flexible and feedback- oriented alternative to manual search and filtering processes. This way, traffic managers adapt and operate more efficiently. The traffic managers will also re- main informed about small disturbances and can act accordingly faster and more efficient. The project was conducted at Gaia Systems AB, Norrköping, Sweden. The project primarily aims to enhance the workflow of traffic managers utiliz- ing Gaia’s existing software for public transport management within Östgöta- trafiken. / Denna avhandling presenterar designen och implementationen av en generativ Artificiell Intelligens (AI)-baserad beslutsstödsgränssnitt applicerad på området för kollektivtrafik, utnyttjande både offline och loggad data från både tidigare händelser och realtidsuppdateringar. AI-assistentsystemet utvecklades med hjälp av Large Language Models (LLM) tillsammans med Retrieval Augmented Generation (RAG) och Function Calling API, tillhandahållet av OpenAI, för att automatisera processen att lägga till kunskap till en LLM. Utmaningar som formatering och omstrukturering av data, datahämtningsmetoder, noggrannhet och latens beaktades. Resultatet är en AI-assistent som kan ha en konversation med användare, svara på frågor om avgångar, ankomster, specifika fordonsturer och andra frågor relevanta inom datamängdens område. AI-assistentsystemet har också utvecklats för att tillhandahålla Client Actions som integreras med användargränssnittet, vilket möjliggör interaktiva element som klickbara länkar för att utlösa relevanta åtgärder baserade på den tillhandahållna innehållet. Olika LLM, inklusive GPT-3.5 och GPT-4 med olika temperaturer, jämfördes och utvärderades med en fördefinierad uppsättning frågor parat med en respektive sanning. Genom att använda en konversationell metod syftar projektet till att effektivisera informationsutvinning från omfattande datamängder och erbjuder ett mer flexibelt och feedbackorienterat alternativ till manuella sök- och filtreringsprocesser. På detta sätt kan trafikledare anpassa sig och arbeta mer effektivt. Trafikledarna kommer också att hållas informerade om mindre störningar och kan agera snabbare och mer effektivt. Projektet genomfördes på Gaia Systems AB, Norrköping, Sverige. Projektet syftar främst till att förbättra arbetsflödet för trafikförvaltare som använder Gaia's befintlig programvara för kollektivtrafikhantering inom Östgötatrafiken.

This study investigates the utilization of Large Language Models for Test Case Generation. The study uses the Large Language model and Embedding model provided by Llama, specifically Llama2 of size 7B, to generate test cases given a defined input. The study involves an implementation that uses customization techniques called Retrieval Augmented Generation (RAG) and Prompt Engineering. RAG is a method that in this study, stores organisation information locally, which is used to create test cases. This stored data is used as complementary data apart from the pre-trained data that the large language model has already trained on. By using this method, the implementation can gather specific organisation data and therefore have a greater understanding of the required domains. The objective of the study is to investigate how AI-driven test case generation impacts the overall software quality and development efficiency. This is evaluated by comparing the output of the AI-based system, to manually created test cases, as this is the company standard at the time of the study. The AI-driven test cases are analyzed mainly in the form of coverage and time, meaning that we compare to which degree the AI system can generate test cases compared to the manually created test case. Likewise, time is taken into consideration to understand how the development efficiency is affected. The results reveal that by using Retrieval Augmented Generationin combination with Prompt Engineering, the system is able to identify test cases to a certain degree. The results show that 66.67% of a specific project was identified using the AI, however, minor noise could appear and results might differ depending on the project’s complexity. Overall the results revealed how the system can positively impact the development efficiency and could also be argued to have a positive effect on the software quality. However, it is important to understand that the implementation as its current stage, is not sufficient enough to be used independently, but should rather be used as a tool to more efficiently create test cases.

With the constant evolution of Large Language Models (LLMs), methods for minimizing hallucinations are being developed to provide more truthful answers. By using Retrieval-Augmented Generation (RAG), external data can be provided to the model on which its answers should be based. This project aims at using RAG for a data extraction pipeline specified for second-life batteries. By pre-defining the prompts the user may only provide the documents that are wished to be analyzed, this is to ensure that the answers are in the correct format for further data processing. To process different document types, initial labeling takes place before more specific extraction suitable for the document can be applied. Best performance is achieved by grouping questions that allow the model to reason around what the relevant questions are so that no hallucinations occur. Regardless of whether there are two or three document types, the model performs equally well, and it is clear that a pipeline of this type is well suited to today's models. Further improvements can be achieved by utilizing models containing a larger context window and initially using Optical Character Recognition (OCR) to read text from the documents.

In recent years, there has been an increase in the issue of bullying, particularly in academic settings. This degree project examines the use of supervised machine learning techniques to identify bullying in text data from school surveys provided by the Friends Foundation. It evaluates various traditional algorithms such as Logistic Regression, Naive Bayes, SVM, Convolutional neural networks (CNN), alongside a Retrieval-Augmented Generation (RAG) model using Llama 3, with a primary goal of achieving high recall on the texts consisting of bullying while also considering precision, which is reflected in the use of the F3-score. The SVM model emerged as the most effective among the traditional methods, achieving the highest F3-score of 0.83. Although the RAG model showed promising recall, it suffered from very low precision, resulting in a slightly lower F3-score of 0.79. The study also addresses challenges such as the small and imbalanced dataset as well as emphasizes the importance of retaining stop words to maintain context in the text data. The findings highlight the potential of advanced machine learning models to significantly assist in bullying detection with adequate resources and further refinement.

Current artificial intelligence tools, including machine learning and large language models, display inabilities to interact with medical data in real time and raise privacy concerns related to user data management. This study illustrates the development of a system prototype using LangChain4j, which is an open-source project offering a multitude of AI-tools, including embedding tools, retrieval-augmented generation, and unified API:s for large language model providers. It was utilized to process medical data from a Neo4j database and enabled real-time interaction for that data. All content generation was generated locally to address privacy concerns, while using Apache Kafka for data distribution. The system prototype was evaluated by response time, resource consumption and accuracy assessment. Among the models assessed, LLaMA 3 emerged as the top performer in accuracy, successfully identifying 42.87% of all attributes with a correctness rate of 89.81%. Meanwhile, Phi3 exhibited superior outcomes in both resource consumption and response time. The embedding process, while enabling the selection of visible data, imposed limitations on general usability. In summary, this thesis advances data interaction using AI by developing a prototype that enables real-time interaction with medical data. It achieves high accuracy and efficient resource utilization while addressing limitations in current AI tools related to real-time processing and privacy concerns. / Nuvarande verktyg för artificiell intelligens, inklusive maskininlärning och stora språkmodeller, visar oförmåga att interagera med medicinska data i realtid och väcker integritetsproblem relaterade till hantering av användardata. Denna studie illustrerar utvecklingen av ett systemprototyp med LangChain4j, ett open-source-projekt som erbjuder en mängd AI-verktyg, inklusive inbäddningsverktyg, retrieval-augmented generation och enhetliga API för leverantörer av stora språkmodeller. Det användes för att bearbeta medicinska data från en Neo4j-databas och möjliggjorde realtidsinteraktion för dessa data. All innehållsgenerering skedde lokalt med Apache Kafka för datadistribution. Systemprototypen utvärderades utifrån svarstid, resursförbrukning och noggrannhetsbedömning. Bland de modeller som utvärderades visade sig LLaMA 3 vara den bästa presteraren i noggrannhet, och identifierade framgångsrikt 42,87 % av alla attribut med en korrekthet på 89,81 %. Samtidigt visade Phi3 överlägsna resultat både i resursförbrukning och svarstid. Inbäddningsprocessen, medan den möjliggjorde valet av synliga data, innebar begränsningar för allmän användbarhet. Sammanfattningsvis förbättrar denna avhandling datainteraktion med AI genom att utveckla en prototyp som möjliggör realtidsinteraktion med medicinska data. Den uppnår hög noggrannhet och effektiv resursanvändning samtidigt som den adresserar begränsningar i nuvarande AI-verktyg relaterade till realtidsbearbetning och integritetsproblem.

Background. New ways of modeling abstract concepts have been enabled due to the recent boom in generative machine learning brought on by transformer architecture. By modeling abstract concepts within high-dimensional vectors their semantic meaning can be inferred and compared, which allows for methods such as embedding-based retrieval and the groundwork for a retrieval-augmented generation. Large language models can augment their parametric generative capabilities by introducing non-parametric information through retrieval processes. Objectives. Previous studies have explored different uses of embedding-based retrieval and retrieval-augmented generation, and this study examines the impact of these methods when used as an aid to support technicians. Methods. By developing and deploying a proof-of-concept system using embedding-based retrieval and retrieval-augmented generation to the Södra ITs service desk, the thesis could monitor system performance. Introducing a system to the service desk that generates instructional solutions to the support tickets and presenting them to the technician. The thesis investigates both systems' perceived performance based on the participating IT technician's input along with the retention of generated solutions and the quality of the solutions. Results. With 75.4% of the systems generated solutions being classified as reasonable solutions to ticket problems the system was deployed to the service desk. After an evaluation period where the technicians had been working with the system, it was shown that the solutions had a retention rate of 38.4%. These results were validated by a survey conducted at the service desk where the inputs were gathered from the technicians, showing a great deal degree of user engagement but a varying opinion on the system's helpfulness. Conclusions. Despite the varying degrees of opinion on the usefulness of the system among the technicians the numbers from the production test show that a significant amount of tickets were solved with the help of the system. Still, there's a huge dependency on seamless integration with the technicians and ticket quality from the requester. / Bakgrund. Nya sätt att modellera abstrakta begrepp har möjliggjorts tack vare den senaste tidens tillväxt inom generativ maskininlärning tack vare transformatorarkitekturen. Genom att modellera abstrakta begrepp i högdimensionella vektorer kan deras semantiska innebörd tolkas och jämföras, vilket möjliggör metoder som inbäddningsbaserad hämtning och grunden för en hämtningsförstärkt generation. Stora språkmodeller kan utvidga sina parametriska generativa förmågor genom att införa icke-parametrisk information genom hämtningsprocesser. Syfte. Tidigare studier har behandlat olika användningsområden för inbäddningsbaserad hämtning och hämtningsförstärkt generering, och i det här examensarbetet undersöks vilken inverkan dessa metoder har när de används som ett hjälpmedel för supporttekniker. Metod. Genom att utveckla och driftsätta ett prototypsystem som använder inbäddningsbaserad hämtning och hämtningsförstärkt generering till Södra ITs servicedesk, kunde examensarbetet övervaka systemets prestanda. Detta genom att införa ett system i servicedesken som genererar instruktionslösningar till supportärendena och presentera dem för teknikern. Examensarbetet undersöker både systemens upplevda prestanda baserat på den deltagande IT-teknikerns synpunkter tillsammans med kvarhållandet av genererade lösningar och kvaliteten på lösningarna. Resultat. Då 75,4% av de systemgenererade lösningarna klassificerades som rimliga för problemen i ärendena driftsattes systemet i servicedesken. Efter en utvärderingsperiod där teknikerna hade arbetat med systemet visade det sig att lösningarna hade en kvarhållningsgrad på 38,4%. Dessa resultat validerades av en undersökning som utförts vid servicedesken där synpunkter samlades in från teknikerna, vilket visade på en hög grad av användarengagemang men en varierande syn på systemets användbarhet. Slutsatser. Trots de varierande synpunkterna på systemets användbarhet bland teknikerna visar siffrorna från produktionstestningen att en betydande mängd ärenden löstes med hjälp av systemet. Dock är man fortfarande mycket beroende av en smidig integration med teknikerna och en god kvalitet på ärendena från beställaren.