Background: Communication plays a crucial role in supply chain management (SCM) as it facilitates the flow of information, materials, and goods across various stages of the supply chain. In the context of supply planning, each planner manages thousands of supply chain entities and spends a lot of time reading and responding to high volumes of emails related to part orders, delays, and backorders that can lead to information overload and hinder workflow and decision-making. Therefore, streamlining communication and enhancing email management are essential for optimizing supply chain efficiency. Objectives: This study aims to create an automated system that can summarize email conversations between planners, suppliers, and other stakeholders. The goal is to increase communication efficiency using Natural Language Processing (NLP) algorithms to extract important information from lengthy conversations. Additionally, the study will explore the effectiveness of using conditional random fields (CRF) to filter out irrelevant content during preprocessing. Methods: We chose four advanced pre-trained abstractive dialogue summarization models, BART, PEGASUS, T5, and CODS, and evaluation metrics, ROUGE and BERTScore, to compare their performance in effectively summarizing our email conversations. We used CRF to preprocess raw data from around 400 planner-supplier email conversations to extract important sentences in a dialogue format and label them with specific dialogue act tags. We then manually summarized the 400 conversations and fine-tuned the four chosen models. Finally, we evaluated the models using ROUGE and BERTScore metrics to determine their similarity to human references. Results: The results show that the performance of the summarization models has significantly improved after fine-tuning the models with domain-specific data. The BART model achieved the highest ROUGE-1 score of 0.65, ROUGE-L score of 0.56, and BERTScore of 0.95 compared to other models. Additionally, CRF-based preprocessing proved to be crucial in extracting essential information and minimizing unnecessary details for the summarization process. Conclusions: This study shows that advanced NLP techniques can make supply chain communication workflows more efficient. The BART-based email summarization tool that we created showed great potential in giving important insights and helping planners deal with information overload.

Les résumés abstraits de dialogues permettent aux gens de comprendre rapidement les aspects clés des conversations dont la synthèse nécessiterait autrement des efforts considérables. Malgré les progrès considérables réalisés par les grands modèles de langage (LLM), même les modèles les plus puissants souffrent encore d’hallucinations lorsqu’ils génèrent des résumés abstraits et ne parviennent pas à couvrir des aspects importants du contenu sous-jacent. En outre, la vérification humaine de la factualité d’un résumé abstrait peut nécessiter un effort considérable. L’un des moyens de minimiser la charge cognitive liée à la vérification de la qualité d’un résumé consiste à faire en sorte que le résumé cite des phrases dans le contenu original. Cependant, il est rare que les ensembles de données de résumés abstraits citent des passages de texte du contenu original. Même les meilleurs LLM ont du mal à effectuer un résumé basé sur des citations. Pour résoudre ce problème, nous créons l’ensemble de données Tweetsumm++, composé de résumés abstraits soutenus par des citations de dialogues entre clients et entreprises sur Twitter. Nous examinons également une méthode d’entraînement et de formulation de problèmes multitâches qui apprend à effectuer conjointement un résumé extractif et un résumé abstractif faisant référence au contenu extrait. Dans notre configuration, le modèle est également chargé d’étiqueter les phrases clés dans des catégories telles que ISSUE, RESOLUTION,WORKAROUND et autres, qui représentent les principaux éléments clés d’un dialogue. Nous explorons l’impact de la mise au point d’un LLM Mixtral open-source pour effectuer un résumé abstractif basé sur des citations et une catégorisation des phrases clés. En outre, étant donné que l’acquisition d’étiquettes pour un tel ensemble de données est coûteuse, nous explorons une nouvelle méthode d’auto-étiquetage basée sur le feedback de l’IA qui bénéficie du format de résumé basé sur les citations et peut améliorer les modèles en ce qui concerne la qualité des citations. / Abstractive summaries of dialogues allow people to quickly understand key aspects of conversations that might otherwise take considerable effort to synthesize. Despite the tremendous progress made by large language models (LLMs), even the most powerful models still suffer from hallucinations when generating abstractive summaries and fail to cover important aspects of the underlying content. Furthermore, human verification of the factuality of an abstractive summary can entail significant effort. One way to minimize the cognitive load of quality checking an abstractive summary is to have the summary cite sentences within the original content. However, it is uncommon for abstractive summarization datasets to cite passages of text from the original content. Even the best LLMs struggle to perform citation-backed summarization. To address this issue, we create the Tweetsumm++ dataset composed of citation-backed abstractive summaries of dialogues between customers and companies on Twitter. We also examine a multi-task problem formulation and training method that learns to jointly perform extractive, and abstractive summarization which reference the extracted content. In our setup, the model is also tasked with tagging key sentences into categories such as ISSUE, RESOLUTION, WORKAROUND, and others that represent the main key elements of a dialogue. We explore the impact of fine-tuning an open-source Mixtral LLM to perform citation-backed abstractive summarization and key sentence categorization. Further, since acquiring labels for such a dataset is costly, we explore a novel self-labeling method based on AI feedback that benefits from the citation-based summarization format and can improve models with respect to citation quality.

The evolution of Language Models (LMs) has enabled building chatbot systems that are capable of human-like dialogues without the need for fine-tuning the chatbot for a specific task. LMs are stateless, which means that a LM-based chatbot does not have a recollection of the past conversation unless it is explicitly included in the input prompt. LMs have limitations in the length of the input prompt, and longer input prompts require more computational and monetary resources, so for longer conversations, it is often infeasible to include the whole conversation history in the input prompt. In this project a short-term memory module is designed and implemented to provide the chatbot context of the past conversation. We are introducing two methods, LimContext method and FullContext method, for producing an abstractive summary of the conversation history, which encompasses much of the relevant conversation history in a compact form that can then be supplied with the input prompt in a resource-effective way. To test these short-term memory implementations in practice, a user study is conducted where these two methods are introduced to 9 participants. Data is collected during the user study and each participant answers a survey after the conversation. These results are analyzed to assess the user experience of the two methods and the user experience between the two methods, and to assess the effectiveness of the prompt design for both answer generation and abstractive summarization tasks. According to the statistical analysis, the FullContext method method produced a better user experience, and this finding was in line with the user feedback. / Utvecklingen av LMs har gjort det möjligt att bygga chatbotsystem kapabla till mänskliga dialoger utan behov av att finjustera chatboten för ett specifikt uppdrag. LMs är stateless, vilket betyder att en chatbot baserad på en LM inte sparar tidigare delar av konversationen om de inte uttryckligen ingår i prompten. LMs begränsar längden av prompten, och längre prompter kräver mer beräknings- och monetära resurser. Således är det ofta omöjligt att inkludera hela konversationshistoriken i prompten. I detta projekt utarbetas och implementeras en korttidsminnesmodul, vars syfte är att tillhandahålla chatboten kontexten av den tidigare konversationen. Vi introducerar två metoder, LimContext metod och FullContext metod, för att ta fram en abstrakt sammanfattning av konversationshistoriken. Sammanfattningen omfattar mycket av det relevanta samtalet i en kompakt form, och kan sedan resurseffektivt förses med den påföljande prompten. För att testa dessa korttidsminnesimplementationer i praktiken genomförs en användarstudie där de två metoderna introduceras för 9-deltagare. Data samlas in under användarstudier. Varje deltagare svarar på en enkät efter samtalet. Resultaten analyseras för att bedöma användarupplevelsen av de två metoderna och användarupplevelsen mellan de två metoderna, och för att bedöma effektiviteten av den snabba designen för både svarsgenerering och abstrakta summeringsuppgifter. Enligt den statistiska analysen gav metoden FullContext metod en bättre användarupplevelse. Detta fynd var även i linje med användarnas feedback.