Natural Language machine learning methods are applied to rules generated to identify malware at the network level. These rules use a computer-based signature specification "language" called Snort. Using Natural Language processing techniques and other machine learning methods, new rules are generated based on a training set of existing Snort rule signatures for a specific type of malware family. The performance is then measured, in terms of the detection of existing types of malware and the number of "false positive" triggering events.

We explore a dataset of almost half a million English song lyrics through three different processes - automatic evaluation, visualization, and generation. We create our own rhyme detector, using the EM algorithm with several improvements and adjustable parameters. This may, in some cases, replace human evaluators that cannot be used, for example, after each iteration of the lyrics generator to evaluate its improvement. By creating a web-page visualization of the results with interesting matrix rhyme highlighting, we make our evaluation accessible to the public. We discuss interesting genre differences discovered by applying our automatic evaluation on the entire dataset. Finally, we explore lyrics generation using state-of-the-art GPT-2.

Generative machine learning models are capable of generating remarkably realistic samples. Some models generate images that look entirely natural, and others generate text that reads as if a human wrote it. However, judging the quality of these models is a major challenge. Today, the most convincing method is to use humans to evaluate the quality of generated samples. However, humans are biased, costly, and inefficient. Therefore, there is a great need for automatic methods. MAUVE is a recent advancement in the evaluation of generative text models. It compares generated data with real data and returns a score that quantifies their similarity. This is accomplished with the help of a neural network, which provides the understanding of text required to evaluate its quality. MAUVE is motivated by its correspondence with human judgment, and this is shown in multiple experiments. This thesis contributes in two significant ways: First, we complement experiments and discussions made in the original paper. Importantly, we demonstrate that MAUVE sometimes fails to recognize quality differences between generative models. This failure is due to the choice of neural network. Later, we demonstrate that MAUVE can be used for more than just text evaluation. Specifically, we show that it can be applied to images. This is accomplished by using a neural network specialized in image recognition. However, the steps can be repeated for any data type, meaning that MAUVE can potentially become a more generalized measurement than suggested in the original paper. Our second contribution is an extension toMAUVEcalled Sequence-MAUVE (S-MAUVE). The score MAUVE produces can be seen as an average of the overall quality of generated text. However, some generative models initially produce excellent text, but see drops in quality as the sequences grow longer. Therefore, a single score that represents entire sequences is likely to omit important details. Instead, S-MAUVE evaluates generated text at the smallest possible level. The result is a sequence of scores, which give users more detailed feedback about the behavior of a generative model. / Generativa maskininlärningsmodeller kan generera data av enastående kvalitet. Vissa modeller genererar bilder av ansikten som ser helt realistiska ut, och andra genererar text som verkar varit skriven av en människa. Trots detta så är det inte klart hur dessa modeller ska evalueras. Idag så är den främsta metoden mänsklig evaluering: En person får utgöra huruvida generade data verkar realistisk eller inte. Mänsklig evaluering har flera nackdelar. Människor är partiska, dyra och långsamma. Därför behövs det automatiska evalueringsverktyg. MAUVE är ett ny metod för att evaluera generative textmodeller som jämför hur lik genererad data är med äkta data. Detta åstadkoms med hjälp av ett neuralt nätverk, som bidrar med den förståelse av text som krävs för att evaluera den. MAUVE är motiverat av att dess omdömen överensstämmer med mänsklig evaluering. Den här uppsatsen bidrar på två sätt. Till att börja med komplementerar vi experiment och diskussioner gjorda i den ursprungliga rapporten o m MAUVE. Till exempel så visar vi att MAUVE ibland inte lyckas känna av kvalitetsskillnader mellan olika generativa modeller. Detta på grund av val av neuralt nätverk. Efteråt så demonstrerar vi att MAUVE kan appliceras på andra typer av data än text. Mer specifikt så applicerar vi MAUVE på bilder. Detta åstadkoms genom att använda ett neuralt nätverk specialiserat på bildigenkänning, istället för text. Stegen vi följer kan upprepas för vilken typ av data som helst, vilket innebär att MAUVE kan användas som ett mer generellt mått än vad den ursprungliga artikeln ger sken för. Vårt andra bidrag är att utveckla MAUVE till det vi kallar för S-MAUVE. MAUVE använder bara sammanfattningar av hela texter som bas för sina jämförelser. En konsekvens av det är att den endast gör påståenden om textdatas genomsnittliga kvalitet. Men, det är välkänt att kvaliteten hos genererad textdata kan variera beroende på var i texten man befinner sig. Många generativa textmodeller producerar sekvenser som är verklighetstrogna i början, men blir sämre och repetitiva senare. Till skillnad från MAUVE så evaluerar S-MAUVE genererad text på minsta möjliga detaljnivå. Resultaten är en sekvens av poäng, som ger användare mer information om egenskaperna hos den studerade generativa modellen.

Deep neural language models like GPT-2 is undoubtedly strong at text generation, but often requires special decoding strategies to prevent producing degenerate output - namely repetition. The use of maximum likelihood training objective results in a peaked probability distribution, leading to the over-confidence of neural networks. In this thesis, we explore entropy regularization for a neural language model that can easily smooth peaked output distribution during the fine-tuning process employing GPT-2. We first define the models in three ways: (1) Out of-the box model without fine-tuning process, (2) Fine-tuned model without entropy regularization, and (3) Fine-tuned model with entropy regularization. To investigate the effect of domains on the model, we also divide the dataset into three ways: (1) fine-tuned on heterogeneous dataset, tested on heterogeneous dataset, (2) fine-tuned on homogeneous dataset, tested on homogeneous dataset, and (3) fine-tuned on heterogeneous dataset, tested on homogeneous dataset. In terms of entropy regularization, we experiment controlling the entropy strength parameter (𝛽) in the range of [0.5, 1.0, 2.0, 4.0, 6.0] and annealing the parameter during fine-tuning process. Our findings prove that the entropy-based regularization during fine-tuning process improve the text generation models by significantly reducing the repetition rate without tuning the decoding strategies. As a result of comparing the probabilities of human-generated sentence tokens, it was observed that entropy regularization compensates for the shortcomings of the deterministic decoding method (Beam search) that mostly selects few high-probability words. Various studies have explored entropy regularization in the cold-start training process of neural networks. However, there are not many studies covering the effect of the fine-tuning stage of text generation tasks when employing large scale pre-trained language models. Our findings present strong evidence that one can achieve significant improvement in text generation by way of utilizing entropy regularization, a highly cost-effective approach, during the fine-tuning process.

Text mining has gained considerable attention due to the extensive usage ofelectronic documents. The significant increase in electronic document usagehas created a necessity to process and analyze them effectively. Rule-basedsystems have traditionally been used to evaluate short pieces of text, but theyhave limitations, including the need for significant manual effort to create andmaintain rules and a high risk of complex bugs. As a result, text classificationhas emerged as a promising solution for extracting meaning from short texts,which are defined as texts limited by a specific character count or word count.This study investigates the feasibility and effectiveness of text classification inclassifying short pieces of text according to their appropriate text properties,based on users’ intentions in the text. The study focuses on comparing twotransformer models, GPT-2 and BERT, in their ability to classify short texts.While other studies have compared these models in intention classificationof text, this study is unique in its examination of their performance onshort pieces of text in this specific context. This study uses user-labelleddata to fine-tune the models, which are then tested on a test dataset fromthe same source. The comparative analysis of the models indicates thatBERT generally outperforms GPT-2 in classifying users’ intentions basedon the appropriate text properties, with an F1-score of 0.68 compared toGPT-2’s F1-score of 0.51. However, GPT-2 performed better on certainclosely related classes, suggesting that both models capture interesting featuresof these classes. Furthermore, the results demonstrated that some classeswere accurately classified despite being context-dependent and positionedwithin longer sentences, indicating that the models likely capture features ofthese classes and facilitate their classification. Both models show promisingpotential as classification models for short texts based on users’ intentions andtheir associated text properties. However, further research may be necessary toimprove their accuracy. Suggestions for enhancing their performance includeutilizing more recent versions of GPT, such as GPT-3 or GPT-4, optimizinghyperparameters, adjusting preprocessing methods, and adopting alternativeapproaches to handle data imbalance. Additionally, testing the models ondatasets from diverse domains with more intricate contexts could providegreater insight into their limitations. / Textutvinning har fått stor uppmärksamhet på grund av den omfattande användningen av elektroniska dokument. Den betydande ökningen av användningen av elektroniska dokument har skapat ett behov av att bearbeta och analysera dem på ett effektivt sätt. Regelbaserade system har traditionellt använts för att utvärdera korta textstycken, men de har begränsningar, bland annat behovet av betydande manuellt arbete för att skapa och upprätthålla regler och en hög risk för komplexa fel. Som ett resultat av detta har textklassificering framstått som en lovande lösning för att utvinna mening ur korta texter, som definieras som texter som begränsas av ett visst antal tecken eller ord. I den här studien undersöks om textklassificering är genomförbar och effektiv när det gäller att klassificera korta textstycken enligt deras lämpliga textegenskaper, baserat på användarnas intentioner i texten. Studien fokuserar på att jämföra två transformatormodeller, GPT-2 och BERT, i deras förmåga att klassificera korta texter. Även om andra studier har jämfört dessa modeller vid avsiktsklassificering av text, är denna studie unik i sin undersökning av deras prestanda för korta textstycken i detta specifika sammanhang. I studien används användarmärkta data för att finjustera modellerna, som sedan testas på ett testdataset från samma källa. Den jämförande analysen av modellerna visar att BERT generellt sett presterar bättre än GPT-2 när det gäller att klassificera användarnas avsikter baserat på lämpliga textegenskaper, med ett F1-värde på 0,68 jämfört med GPT-2:s F1-värde på 0,51. GPT-2 presterade dock bättre på vissa närbesläktade klasser, vilket tyder på att båda modellerna fångar intressanta egenskaper hos dessa klasser. Dessutom visade resultaten att vissa klasser klassificerades korrekt trots att de var kontextberoende och placerade i längre meningar, vilket tyder på att modellerna sannolikt fångar upp egenskaper hos dessa klasser och underlättar deras klassificering. Båda modellerna visar lovande potential som klassificeringsmodeller för korta texter baserade på användarnas intentioner och deras tillhörande textegenskaper. Ytterligare forskning kan dock vara nödvändig för att förbättra deras noggrannhet. Förslag för att förbättra deras prestanda är bland annat att använda nyare versioner av GPT, till exempel GPT-3 eller GPT-4, optimera hyperparametrar, justera förbehandlingsmetoder och anta alternativa metoder för att hantera obalans i data. Om modellerna dessutom testas på dataset från olika områden med mer komplicerade sammanhang kan man få en bättre insikt i deras begränsningar.

Every day, machine learning (ML) and artificial intelligence (AI) embeds itself further into domestic and industrial technologies. Interaction de- signers have historically struggled to engage directly with the subject, facing a shortage of appropriate methods and abstractions. There is a need to find ways though which interaction design practitioners might integrate ML into their work, in order to democratize and diversify the field. This thesis proposes a mode of inquiry that considers the inter- active qualities of what machine learning does, as opposed the tech- nical specifications of what machine learning is. A shift in focus from the technicality of ML to the artifacts it creates allows the interaction designer to situate its existing skill set, affording it to engage with ma- chine learning as a design material. A Research-through-Design pro- cess explores different methodological adaptions, evaluated through user feedback and an-in depth case analysis. An elaborated design experiment, Multiverse, examines the novel, non-anthropomorphic aesthetic qualities of generative literature. It prototypes interactions with bidirectional literature and studies how these transform the reader into a cybertextual “user-reader”. The thesis ends with a discussion on the implications of machine written literature and proposes a number of future investigations into the research space unfolded through the prototype.

The availability of the stenographic manuscripts of Astrid Lindgren have sparked an interest in the creation of a language model for stenography. By its very nature stenography is low-resource and the unavailability of data requires a tool for using normal data. The tool presented in this thesis is to create stenographic data from manipulating orthographic data. Stenographic data is distinct from orthographic data through three different types manipulations that can be carried out. Firstly stenography is based on a phonetic version of language, secondly it used its own alphabet that is distinct from normal orthographic data, and thirdly it used several techniques to compress the data. The first type of manipulation is done by using a grapheme-to-phoneme converter. The second type is done by using an orthographic representation of a stenographic alphabet. The third type of manipulation is done by manipulating based on subword level, word level and phrase level. With these manipulations different datasets are created with different combinations of these manipulations. Results are measured for both perplexity on a GPT-2 language model and for compression rate on the different datasets. These results show a general decrease of perplexity scores and a slight compression rate across the board. We see that the lower perplexity scores are possibly due to the growth of ambiguity.