The emergence of large language models, such as BERT and GPT-3, has revolutionized natural language processing tasks. However, the development and deployment of these models pose challenges, including concerns about computational resources and environmental impact. This study aims to compare discriminative language models for text classification based on their performance and usage cost. We evaluate the models using a hierarchical multi-label text classification task and assess their performance using primarly F1-score. Additionally, we analyze the usage cost by calculating the Floating Point Operations (FLOPs) required for inference. We compare a baseline model, which consists of a classifier chain with logistic regression models, with fine-tuned discriminative language models, including BERT with two different sequence lengths and DistilBERT, a distilled version of BERT. Results show that the DistilBERT model performs optimally in terms of performance, achieving an F1-score of 0.56 averaged on all classification layers. The baseline model and BERT with a maximal sequence length of 128 achieve F1-scores of 0.51. However, the baseline model outperforms the transformers at the most specific classification level with an F1-score of 0.33. Regarding usage cost, the baseline model significantly requires fewer FLOPs compared to the transformers. Furthermore, restricting BERT to a maximum sequence length of 128 tokens instead of 512 sacrifices some performance but offers substantial gains in usage cost. The code and dataset are available on GitHub. / Uppkomsten av stora språkmodeller, som BERT och GPT-3, har revolutionerat språkteknologi. Dock ger utvecklingen och implementeringen av dessa modeller upphov till utmaningar, bland annat gällande beräkningsresurser och miljöpåverkan. Denna studie syftar till att jämföra diskriminativa språkmodeller för textklassificering baserat på deras prestanda och användningskostnad. Vi utvärderar modellerna genom att använda en hierarkisk textklassificeringsuppgift och bedöma deras prestanda primärt genom F1-score. Dessutom analyserar vi användningskostnaden genom att beräkna antalet flyttalsoperationer (FLOPs) som krävs för inferens. Vi jämför en grundläggande modell, som består av en klassifikationskedja med logistisk regression, med finjusterande diskriminativa språkmodeller, inklusive BERT med två olika sekvenslängder och DistilBERT, en destillerad version av BERT. Resultaten visar att DistilBERT-modellen presterar optimalt i fråga om prestanda och uppnår en genomsnittlig F1-score på 0,56 för alla klassificeringsnivåer. Den grundläggande modellen och BERT med en maximal sekvenslängd på 128 uppnår ett F1-score på 0,51. Dock överträffar den grundläggande modellen transformermodellerna på den mest specifika klassificeringsnivån med en F1-score på 0,33. När det gäller användningskostnaden kräver den grundläggande modellen betydligt färre FLOPs jämfört med transformermodellerna. Att begränsa BERT till en maximal sekvenslängd av 128 tokens ger vissa prestandaförluster men erbjuder betydande besparingar i användningskostnaden. Koden och datamängden är tillgängliga på GitHub.

This study aims to construct a state-of-the-art classifier model and compare it against a largelanguage model. A variation of SVM called LinearSVC was utilised and the BERT model usingbert-base-uncased was used. The data, provided by the Swedish Research Council, consisted ofresearch grant applications. The research grant applications were divided into two groups, whichwere further divided into several subgroups. The subgroups represented research fields such ascomputer science and applied physics. Significant class imbalances were present, with someclasses having only a tenth of the applications of the largest class. To address these imbalances,a new dataset was created using data that had been randomly oversampled. The models weretrained and tested on their ability to correctly assign a subgroup to a research grant application.Results indicate that the BERT model outperformed the SVM model on the original dataset,but not on the balanced dataset . Furthermore, the BERT model’s performance decreased whentransitioning from the original to the balanced dataset, due to overfitting or randomness. / Denna studie har som mål att bygga en state-of-the-art klassificerar model och sedan jämföraden mot en stor språkmodel. SVM modellen var en variation av SVM vid namn LinearSVC ochför BERT användes bert-base-uncased. Data erhölls från Vetenskapsrådet och bestod av forskn-ingsbidragsansökningar. Forskningsbidragsansökningarna var uppdelade i två grupper, som varytterligare uppdelade i ett flertal undergrupper. Dessa undergrupper representerar forsknings-fält såsom datavetenskap och tillämpad fysik. I den data som användes i studien fanns storaskillnader mellan klasserna, där somliga klasser hade en tiondel av ansökningarna som de storaklasserna hade. I syfte att lösa dessa klassbalanseringsproblem skapades en datamängd somundergått slumpmässig översampling. Modellerna tränades och testades på deras förmåga attkorrekt klassificera en forskningsbidragsansökan in i rätt undergrupp. Studiens fynd visade attBERT modellen presterade bättre än SVM modellen på både den ursprungliga datamängden,dock inte på den balanserade datamängden. Tilläggas kan, BERTs prestanda sjönk vid övergångfrån den ursprungliga datamängden till den balanserade datamängden, något som antingen berorpå överanpassning eller slump.

Submitted by Marilene Donadel (marilene.donadel@unioeste.br) on 2019-01-23T18:30:18Z No. of bitstreams: 1 Renata_Saldanha_2018.pdf: 810101 bytes, checksum: b45ce79ad809216543b1f4f2228e57f1 (MD5) / Made available in DSpace on 2019-01-23T18:30:18Z (GMT). No. of bitstreams: 1 Renata_Saldanha_2018.pdf: 810101 bytes, checksum: b45ce79ad809216543b1f4f2228e57f1 (MD5) Previous issue date: 2018-08-31 / This dissertation aims to analyze Restorative Justice and its practices, to find a meeting point for the foundation of these practices in Philosophy, especially based on the systemic-phenomenological theory of Bert Hellinger. Restorative Justice is a relatively new topic in Brazil and it has been increasingly used, but it is still needy the study of this subject when is not under a practical bias. Thus, this work seeks to conceptualize the theme based on the bibliographical review on the subject, with Kant, Hegel and Bert Hellinger. In the first chapter, the context of the flowering of restorative practices in Brazil, with a focus on the criminal area and the essentiality of its theory, is worked on: new vision of conflict, inclusion, participation, (co) responsibility, voluntariness, honesty, humility, interconnection, empowerment, hope, solidarity and the encounter. In the second chapter, Restorative Justice is approached from a critical perspective, especially on the basis of Kant and Hegel, the main framers of the current model of retributive justice.For Kant, crime is the non-fulfillment of a duty and punishment is a punishment for such an action, that is, punishment is the retribution of the evil of crime with the evil of pen, in a strictly formal paradigm. In Hegel, law is the most accurate form of law and its violation hurts the highest degree of human freedom. The Law defines the duties and the rights of the subjects. Duty is negative determination and right is positive determination of freedom. But since law and duty can be denied, law internalizes its own negation, so that this negation is not formally infinite. Thus, the denial of law by the law itself is the sanction, which also denotes a formalist bias of the concept of justice and punishment. Finally, in the last chapter, and after locating the central elements of restorative practices, we seek in Bert Hellinger's systemic-phenomenological theory a foundation for restorative practices. Bert Hellinger supposes that there are three laws that govern all human relationships: belonging, hierarchy and balance. As every system values inclusiveness, belonging is the right of everyone to be part of it. Hierarchy is the order of precedence of people as time passes. Finally, balance is the trade-off between giving and taking, representing a flow of exchange that animates human relationships. The major point of contact between restorative practices and the systemic-phenomenological theory is the change of perception in relation to the conflict, with the inclusion, which derives from the right to belong, the equality, the dignity of the human person, which makes reconciliation possible and opens the way to peace, enabling, in turn, the construction of the sense of justice. concluding that Restorative Justice is a meeting with itself and with the other, face-to-face, aiming to understand the hidden causes and entanglements which led to conflict in a larger context (beyond the conflict), with the assumption of the responsibility of each one to the event of the conflict and construction of the systemic reparation of damages (material, spiritual, emotional, transgenerational, psychological, symbolic). Bert Hellinger's theory allows us to transcend the differentiations that exclude and restore the basic human need for connection with other human beings. / Esta dissertação tem por objetivo analisar a Justiça Restaurativa e suas práticas e encontrar um ponto de encontro para a fundamentação destas práticas na Filosofia, especialmente com base na teoria sistêmico-fenomenológica de Bert Hellinger. A Justiça Restaurativa é um tema relativamente novo no Brasil e ela vem sendo cada vez mais utilizada, mas ainda é carente o estudo desse tema que não seja sob um viés prático. Assim, este trabalho busca conceituar o tema com base na revisão bibliográfica sobre o assunto, com apoio na filosofia de Kant, Hegel e Bert Hellinger. No primeiro capítulo, é trabalhado o contexto de florescimento das práticas restaurativas no Brasil, com enfoque na área criminal e a essencialidade de sua teoria: nova visão do conflito, inclusão, participação, (co)responsabilidade, voluntariedade, honestidade, humildade, interconexão, empoderamento, esperança, solidariedade e o encontro. No segundo capítulo, a Justiça Restaurativa é abordada sob uma perspectiva crítica, especialmente com base em Kant e Hegel, principais estruturadores do modelo de justiça retributivo vigente. Para Kant, o crime é o descumprimento de um dever e a punição é um castigo para tal ação, ou seja, a punição é a retribuição do mal do crime com o mal da pena, em um paradigma estritamente formal. Em Hegel, a lei constitui a forma mais apurada do Direito e sua violação fere o mais alto grau da liberdade humano. O Direito define os deveres e os direitos dos sujeitos. O dever é determinação negativa e o direito é determinação positiva da liberdade. Mas como o direito e o dever podem ser negados, o Direito interioriza sua própria negação, a fim de que essa negação não seja formalmente infinita. Assim, a negação do Direito pelo próprio Direito é a sanção, o que denota também um viés formalista do conceito de Justiça e punição. Por fim, no último capítulo, e após situar os elementos centrais das práticas restaurativas, busca-se na teoria sistêmico-fenomenológica de Bert Hellinger uma fundamentação para as práticas restaurativas. Bert Hellinger supõe que existem três leis que regem todos os relacionamentos humanos: o pertencimento, a hierarquia e o equilíbrio. Como todo sistema preza pela inclusão, o pertencimento é o direito de todos de fazerem parte. A hierarquia é a ordem de precedência das pessoas conforme o passar do tempo. Por fim, o equilíbrio é a compensação entre o dar e o tomar, representando um fluxo de troca que anima as relações humanas. O maior ponto de contato entre as práticas restaurativas e a teoria sistêmico-fenomenológica é a mudança de percepção em relação ao conflito, com a inclusão, que decorre do direito de pertencer, a igualdade, a dignidade da pessoa humana, o que possibilita a reconciliação e abre o caminho para a paz, possibilitando, por sua vez, a construção do sentido de Justiça. A Justiça Restaurativa assim representa um encontro consigo próprio e com o outro, face-a-face, visando compreender as causas ocultas e emaranhamentos que levaram ao conflito diante de um contexto maior (para além do conflito), com a assunção da responsabilidade de cada um para o acontecimento do conflito e construção da reparação sistêmica dos danos (material, espiritual, emocional, transgeracional, psicológico, simbólico). A teoria de Bert Hellinger permite transcender as diferenciações que excluem e restaurar a necessidade humana básica de conexão com os demais seres humanos.

Pharmacovigilance, also referred to as drug safety, is an important science for identifying risks related to medicine intake. Side effects of medicine can be caused by for example interactions, high dosage and misuse. In order to find patterns in what causes the unwanted effects, information needs to be gathered and mapped to predefined terms. This mapping is today done manually by experts which can be a very difficult and time consuming task. In this thesis the aim is to automate the process of mapping side effects by using machine learning techniques. The model was developed using information from preexisting mappings of verbatim expressions of side effects. The final model that was constructed made use of the pre-trained language model BERT, which has received state-of-the-art results within the NLP field. When evaluating on the test set the final model performed an accuracy of 80.21%. It was found that some verbatims were very difficult for our model to classify mainly because of ambiguity or lack of information contained in the verbatim. As it is very important for the mappings to be done correctly, a threshold was introduced which left for manual mapping the verbatims that were most difficult to classify. This process could however still be improved as suggested terms were generated from the model, which could be used as support for the specialist responsible for the manual mapping.

The goal of this project was to create a chatbot capable of debating a user using limited resources including a discussion thread from the online debate forum Kialo. A retrieval based bot was designed and the discussion thread was converted into a database which the bot could interpret and choose an appropriate answer from. Which answer is appropriate is decided by the bot using a few key features in a given input sentence. The main features are word similarity, sentiment distance and BERT-encoding (a model for vector representation of text created by Google). The similarity of these features where then used to score claims from the dataset. Combining and weighting the scores was then used to find the correct response to a given input sentence. The most successful of the features was BERT-encoding. Once the bot had been refined it was brought online and tested using the communication platform Discord.

Promoting constructiveness in online comment sections is an essential step to make the internet a more productive place. On online marketplaces, customers often have the opportunity to voice their opinion and relate their experience with a given product. In this thesis, we investigate the possibility to model constructiveness in product review in order to promote the most informative and argumentative customer feedback. We develop a new constructiveness 4-class scale taxonomy based on heuristics and specific categorical criteria. We use this taxonomy to annotate 4000 Amazon customer reviews as our training set, referred to as the Corpus for Review Constructiveness (CRC). In addition to the 4-class constructiveness tag, we include a binary tag to compare modeling performance with previous work. We train and test several computational models such as Bidirectional Encoder Representations from Transformers (BERT), a Stacked Bidirectional LSTM and a Gradient Boosting Machine. We demonstrate our annotation scheme’s reliability with a set of inter-annotator agreement experiments, and show that good levels of performance can be reached in both multiclass setting (0.69 F1 and 57% error reduction over the baseline) and binary setting (0.85 F1 and 71% error reduction). Different features are evaluated individually and in combination. Moreover, we compare the advantages, downsides and performance of both feature-based and neural network models. Finally, these models trained on CRC are tested on out-of-domain data (news article comments) and shown to be nearly as proficient as on in-domain data. This work allows the extension of constuctiveness modeling to a new type of data and provides a new non-binary taxonomy for data labeling.

In this thesis, we show how a text-based Recommendation Systems can greatly benefit from neural statistical language models, more particularly BERT. We evaluate the framework on a digital and collaborative platform for radiologists, by automatically suggesting scientific papers from the medical database PubMed, to provide evidence in diagnostic radiology. The models use contextualized vectors to represent text, accounting for writing style, misspelling and jargon. By using pre-computed representations of text passages, we are able to use compute-heavy statistical language models in production environments, where supercomputers are not available during inference. The results suggest pre-computed embeddings are very effective when the texts came from the same domain, and less effective (but still useful) in capturing the interaction between clinical and scientific text. Nonetheless, the suggested solutions hold promises in this and other areas in medicine. Possibly, the results are transferable to other domains, such as processing of legal documents and patent search.

The way people acquire knowledge has largely shifted from print to web resources. Meanwhile, search has become the main medium to access information. Amongst various search behaviors, exploratory search represents a learning process that involves complex cognitive activities and knowledge acquisition. Research on exploratory search studies on how to make search systems help people seek information and develop intellectual skills. This research focuses on information retrieval and aims to build an exploratory search system that shows higher clustering performance and diversified search results. In this study, a new language model that integrates the state-of-the-art vector language model (i.e., BERT) with human knowledge is built to better understand and organize search results. The clustering performance of the new model (i.e., RDF+BERT) was similar to the original model but slight improvement was observed with conversational texts compared to the pre-trained language model and an exploratory search baseline. With the addition of the enrichment phase of expanding search results to related documents, the novel system also can display more diverse search results.

Polestar has during the last couple of months more than doubled its amount of customer feedback, and the forecast for the future is that this amount will increase even more. Manually reading this feedback is expensive and time-consuming, and for this reason there's a need to automatically analyse the customer feedback. The company wants to understand the customer and extract trends and topics that concerns the consumer in order to improve the customer experience. Over the last couple of years as Natural Language Processing developed immensely, new state of the art language models have pushed the boundaries in all type of benchmark tasks. In this thesis have three different extractive summarization models and three different keyword extraction methods been tested and evaluated based on two different quantitative measures and human evaluation to extract information from text. This master thesis has shown that extractive summarization models with a Transformer-based text representation are best at capturing the context in a text. Based on the quantitative results and the company's needs, Textrank with a Transformer-based embedding was chosen as the final extractive summarization model. For Keywords extraction was the best overall model YAKE!, based on the quantitative measure and human validation

Eftersom mental ohälsa stiger i samhället och forskningen inte har ett tydligt svar så har vi i detta projekt formulerat en egen hypotes om varför vi ser den här trenden. Eftersom nyhetstjänster tjänar på att publicera negativa artiklar så leder det till att fler konsumerar negativa nyheter. Målet med projektet är att ta fram en nyhetsaggregator som utför sentimentanalys påaktuella nyheter från Aftonbladet, Expressen, DN och SVT där nyheterna kategoriseras i positiva, neutrala och negativa nyheter. Nyheterna samlas in med en egenutvecklad webskrapare som hämtar nyheterna från respektive källa. Sedan laddas nyheterna upp på en databas och bearbetas sedan för maskininlärning. För klassificering av nyhetsartiklar har vi tränat ett neuralt nätverk som utför klassificering av nyheter i det allmänna nyhetsflödet. Vi har även utvecklat en egen lexikonbaserad modell som är unik för varje användare för att kunna predicera användarspecifika sentiment. Resultat är en egendesignad hemsida med ett allmänt nyhetsflöde, samt ett anpassat flöde för registrerade användare, där man med ett reglage kan reglera vilken typ av nyheter och från vilka nyhetssajter som man vill se nyheter. På hemsidan presenteras även statistik över bland annat hur fördelningen av positiva, neutrala och negativa nyheter är på de olika nyhetssajterna