No description available.

No description available.

<p>    </p> <p>Over the last decade, measurement technology that records neural activity such as ECoG and Utah array has dramatically improved. These advancements have given researchers access to recordings from multiple neurons simultaneously. Efficient computational and statistical methods are required to analyze this data type successfully. The time-series model is one of the most common approaches for analyzing this data type. Unfortunately, even with all the advances made with time-series models, it is not always enough since these models often need massive amounts of data to achieve good results. This is especially true in the field of neuroscience, where the datasets are often limited, therefore imposing constraints on the type and complexity of the models we can use. Not only that, but the Signal-to- noise ratio tends to be lower than in other machine learning datasets. This paper will introduce different architectures and techniques to overcome constraints imposed by these small datasets. There are two major experiments that we will discuss. (1) We will strive to develop models for participants who lost the ability to speak by building upon the previous state-of-the-art model for decoding neural activity (ECoG data) into English text. (2) We will introduce two new models, RNNF and Neural RoBERTa. These new models impute missing neural data from neural recordings (Utah arrays) of monkeys performing kinematic tasks. These new models with the help of novel data augmentation techniques (dynamic masking) outperformed state-of-the-art models such as Neural Data Transformer (NDT) in the Neural Latents Benchmark competition. </p>

The item cold-start problem, which describes the difficulty of recommendation systems in recommending new items to users, remains a great challenge for recommendation systems that rely on past user-item interaction data. A popular technique in the current research surrounding the cold-start problem is the use of hybrid models that combine two or more recommendation strategies that may contribute with their individual advantages. This thesis investigates the use of a hybrid model which combines Sentence BERT embeddings with a recommendation model based on Factorization Machines (FM). The research question is stated as: How does a hybrid recommendation system based on Factorization Machines with frozen Sentence BERT embeddings perform in terms of solving the cold-start problem?. Three experiments were conducted to answer the research question. These involved finding an optimal pre-trained Sentence BERT model, investigating the difference in performance between an FM-model and a hybrid FM-model, as well as the difference in ranking of an item depending on whether or not the hybrid FM-model has been trained on the item. The results show that the best pre-trained Sentence BERT model for producing meaningful embeddings is the paraphrase-MiniLM-L3-v2 model, that a hybrid FM-model and a standard FM-model perform almost equally in terms of precision and recall at 50, and that there is a weak correlation between the item-frequency and how the hybrid FM-model ranks an item when trained and not trained on the item. The answer to the research question is that a recommendation model based on Factorization Machines with frozen Sentence BERT embeddings displays low precision at 50 and recall at 50 values with the given parameters in comparison to the values given in an optimal recommendation scenario. The hybrid FM-model shows cold-start potential due to displaying similar results to the standard FM-model, but these values are so low that further investigation with other parameters is needed for a clearer conclusion. / Kallstartsproblem för artiklar som beskriver svårigheten hos rekommendationssystem gällande uppgiften att rekommendera nya artiklar till användare, är fortsatt en stor utmaning för rekommendationssystem som förlitar sig på data som beskriver interaktioner mellan användare och artiklar. En populär teknik inom den aktuella forskningen gällande kallstartsproblemet är användandet av hybridmodeller som kombinerar två eller flera rekommendationsstrategier och som potentiellt kan bidra med sina individuella fördelar. Detta examensarbete undersöker användandet av en hybridmodell som kombinerar menings-BERT inbäddningar med en rekommendationsmodell baserad på faktoriseringsmaskiner (FM). Frågeställningen lyder: Hur väl kan kallstartsproblemet för rekommendationer lösas med en hybridmodell baserad på faktoriseringsmaskiner med frusna menings-BERT-inbäddningar?. Tre experiment utfördes för att svara på frågeställningen. Dessa experiment innebar att hitta en optimal förtränad menings-BERT-modell, undersöka skillnaden i utförandet mellan en FM-modell och en hybrid FM-modell, samt skillnaden i ranking av en artikel baserat på huruvida hybridmodellen tränats eller inte tränats på artikeln. Resultaten visar att den bästa förtränade menings-BERT-modellen gällande skapandet av meningsfulla inbäddningar är paraphrase-MiniLM-L3-v2-modellen, att en hybrid FM-modell och en FM-modell genererar nästan identiska resultat baserat på precision och återkallelse för de första 50 resultaten och att det finns en svag korrelation mellan artikel-frekvens och hur hybridmodellen rankar en artikel när hybridmodellen tränats eller inte tränats på artikeln. Svaret på frågeställningen är att en hybrid FM-modell med frusna menings-BERT-inbäddningar visar låga resultat för precision och återkallelse för de första 50 resultaten givet de använda parametrarna jämfört med de värden som skulle genererats i ett optimalt rekommendationsscenario. Den hybrida FM-modellen visar kallstartspotential då den visar liknande resultat som FM-modellen, men dessa värden är så låga att frågan behöver undersökas ytterligare för tydligare resultat.

School teachers spend approximately 30 percent of their time grading exams and other assessments. With an increasingly digitized education, a research field have been initiated that aims to reduce the time spent on grading by automating it. This is an easy task for multiple-choice questions but much harder for open-ended questions requiring free-text answers, where the latter have shown to be superior for knowledge assessment and learning consolidation. While results in previous work have presented promising results of up to 90 percent grading accuracy, it is still problematic using a system that gives the wrong grade in 10 percent of the cases. This has given rise to a research field focusing on assisting teachers in the grading process, instead of fully replacing them. Cluster analysis has been the most popular tool for this, grouping similar answers together and letting teachers process groups of answers at once, instead of evaluating each question one-at-a-time. This approach has shown evidence to decrease the time spent on grading substantially, however, the methods for performing the clustering vary widely between studies, leaving no apparent methodology choice for real-use implementation. Using several techniques for pre-processing, text representation and choice of clustering algorithm, this work compared various methods for clustering free-text answers by evaluating them on a dataset containing almost 400 000 student answers. The results showed that using all of the tested pre-processing techniques led to the best performance, although the difference to using minimum pre-processing were small. Sentence embeddings were the text representation approach that performed the best, however, it remains to be answered how it should be used when spelling and grammar is part of the assessment, as it lacks the ability to identify such errors. A suitable choice of clustering algorithm is one where the number of clusters can be specified, as determining this automatically proved to be difficult. Teachers can then easily adjust the number of clusters based on their judgement. / Skollärare spenderar ungefär 30 procent av sin tid på rättning av prov och andra bedömningar. I takt med att mer utbildning digitaliseras, försöker forskare hitta sätt att automatisera rättning för att minska den administrativa bördan för lärare. Flervalsfrågor har fördelen att de enkelt kan rättas automatiskt, medan öppet ställda frågor som kräver ett fritt formulerat svar har visat sig vara ett bättre verktyg för att mäta elevers förståelse. Dessa typer av frågor är däremot betydligt svårare att rätta automatiskt, vilket lett till forskning inom automatisk rättning av dessa. Även om tidigare forskning har lyckats uppnå resultat med upp till 90 procents träffsäkerhet, är det fortfarande problematiskt att det blir fel i de resterande 10 procenten av fallen. Detta har lett till forskning som fokuserar på underlätta för lärare i rättningen, istället för att ersätta dem. Klusteranalys har varit det mest populära tillvägagångssättet för att åstadkomma detta, där liknande svar grupperas tillsammans, vilket möjliggör rättning av flera svar samtidigt. Denna metod har visat sig minska rättningstiden signifikant, däremot har metoderna för att göra klusteranalysen varierat brett, vilket gör det svårt att veta hur en implementering i ett verkligt scenario bör se ut. Genom att använda olika tekniker för textbearbetning, textrepresentation och val av klusteralgoritm, jämför detta arbete olika metoder för att klustra fritext-svar, genom att utvärdera dessa på nästan 400 000 riktiga elevsvar. Resultatet visar att mer textbearbetning generellt är bättre, även om skillnaderna är små. Användning av så kallade sentence embeddings ledde till bäst resultat när olika tekniker för textrepresentation jämfördes. Däremot har denna teknik svårare att identifiera grammatik- och stavningsfel, hur detta ska hanteras är en fråga för framtida forskning. Ett lämpligt val av klustringsalgoritm är en där antalet kluster kan bestämmas av användaren, då det visat sig svårt att bestämma det automatiskt. Lärare kan då justera antalet kluster ifall det skulle vara för få eller för många.

My dissertation is about how engineering educators can use natural language processing (NLP) in implementing open-ended assessments in undergraduate engineering degree programs. Engineering students need to develop an ability to exercise judgment about better and worse outcomes of their decisions. One important consideration for improving engineering students' judgment involves creating sound educational assessments. Currently, engineering educators face a trad-off in selecting between open- and closed-ended assessments. Closed-ended assessments are easy to administer and score but are limited in what they measure given students are required, in many instances, to choose from a priori list. Conversely, open-ended assessments allow students to write their answers in any way they choose in their own words. However, open-ended assessments are likely to take more personal hours and lack consistency for both inter-grader and intra-grader grading. The solution to this challenge is the use of NLP. The working principles of the existing NLP models is the tallying of words, keyword matching, or syntactic similarity of words, which have often proved too brittle in capturing the language diversity that students could write. Therefore, the problem that motivated the present study is how to assess student responses based on underlying concepts and meanings instead of morphological characteristics or grammatical structure in sentences. Some of this problem can be addressed by developing NLP-assisted grading tools based on transformer-based large language models (TLLMs) such as BERT, MPNet, GPT-4. This is because TLLMs are trained on billions of words and have billions of parameters, thereby providing capacity to capture richer semantic representations of input text. Given the availability of TLLMs in the last five years, there is a significant lack of research related to integrating TLLMs in the assessment of open-ended engineering case studies. My dissertation study aims to fill this research gap. I developed and evaluated four NLP approaches based on TLLMs for thematic analysis of student responses to eight question prompts of engineering ethics and systems thinking case scenarios. The study's research design comprised the following steps. First, I developed an example bank for each question prompt with two procedures: (a) human-in-the-loop natural language processing (HILNLP) and (b) traditional qualitative coding. Second, I assigned labels using the example banks to unlabeled student responses with the two NLP techniques: (i) k-Nearest Neighbors (kNN), and (ii) Zero-Shot Classification (ZSC). Further, I utilized the following configurations of these NLP techniques: (i) kNN (when k=1), (ii) kNN (when k=3), (iii) ZSC (multi-labels=false), and (iv) ZSC (multi-labels=true). The kNN approach took input of both sentences and their labels from the example banks. On the other hand, the ZSC approach only took input of labels from the example bank. Third, I read each sentence or phrase along with the model's suggested label(s) to evaluate whether the assigned label represented the idea described in the sentence and assigned the following numerical ratings: accurate (1), neutral (0), and inaccurate (-1). Lastly, I used those numerical evaluation ratings to calculate accuracy of the NLP approaches. The results of my study showed moderate accuracy in thematically analyzing students' open-ended responses to two different engineering case scenarios. This is because no single method among the four NLP methods performed consistently better than the other methods across all question prompts. The highest accuracy rate varied between 53% and 92%, depending upon the question prompts and NLP methods. Despite these mixed results, this study accomplishes multiple goals. My dissertation demonstrates to community members that TLLMs have potential for positive impacts on improving classroom practices in engineering education. In doing so, my dissertation study takes up one aspect of instructional design: assessment of students' learning outcomes in engineering ethics and systems thinking skills. Further, my study derived important implications for practice in engineering education. First, I gave important lessons and guidelines for educators interested in incorporating NLP into their educational assessment. Second, the open-source code is uploaded to a GitHub repository, thereby making it more accessible to a larger group of users. Third, I gave suggestions for qualitative researchers on conducting NLP-assisted qualitative analysis of textual data. Overall, my study introduced state-of-the-art TLLM-based NLP approaches to a research field where it holds potential yet remains underutilized. This study can encourage engineering education researchers to utilize these NLP methods that may be helpful in analyzing the vast textual data generated in engineering education, thereby reducing the number of missed opportunities to glean information for actors and agents in engineering education. / Doctor of Philosophy / My dissertation is about how engineering educators can use natural language processing (NLP) in implementing open-ended assessments in undergraduate engineering degree programs. Engineering students need to develop an ability to exercise judgment about better and worse outcomes of their decisions. One important consideration for improving engineering students' judgment involves creating sound educational assessments. Currently, engineering educators face a trade-off in selecting between open- and closed-ended assessments. Closed-ended assessments are easy to administer and score but are limited in what they measure given students are required, in many instances, to choose from a priori list. Conversely, open-ended assessments allow students to write their answers in any way they choose in their own words. However, open-ended assessments are likely to take more personal hours and lack consistency for both inter-grader and intra-grader grading. The solution to this challenge is the use of NLP. The working principles of the existing NLP models are the tallying of words, keyword matching, or syntactic similarity of words, which have often proved too brittle in capturing the language diversity that students could write. Therefore, the problem that motivated the present study is how to assess student responses based on underlying concepts and meanings instead of morphological characteristics or grammatical structure in sentences. Some of this problem can be addressed by developing NLP-assisted grading tools based on transformer-based large language models (TLLMs). This is because TLLMs are trained on billions of words and have billions of parameters, thereby providing capacity to capture richer semantic representations of input text. Given the availability of TLLMs in the last five years, there is a significant lack of research related to integrating TLLMs in the assessment of open-ended engineering case studies. My dissertation study aims to fill this research gap. The results of my study showed moderate accuracy in thematically analyzing students' open-ended responses to two different engineering case scenarios. My dissertation demonstrates to community members that TLLMs have potential for positive impacts on improving classroom practices in engineering education. This study can encourage engineering education researchers to utilize these NLP methods that may be helpful in analyzing the vast textual data generated in engineering education, thereby reducing the number of missed opportunities to glean information for actors and agents in engineering education.

<p>Deep Neural Networks (DNNs) are constantly evolving, enabling the power of deep learning to be applied to an ever-growing range of applications, such as Natural Language Processing (NLP), recommendation systems, graph processing, etc. However, these emerging neural workloads present large computational demands for both training and inference. In this dissertation, we propose optimizations that take advantage of the unique characteristics of different emerging workloads to simultaneously improve accuracy and computational efficiency.</p> <p><br></p> <p>First, we consider Language Models (LMs) used in NLP. We observe that the design process of LMs (pre-train a foundation model, and subsequently fine-tune it for different downstream tasks) leads to models that are highly over-parameterized for the downstream tasks. We propose AxFormer, a systematic framework that applies accuracy-driven approximations to create accurate and efficient LMs for a given downstream task. AxFormer eliminates task-irrelevant knowledge, and helps the model focus only on the relevant parts of the input.</p> <p><br></p> <p>Second, we find that during fine-tuning of LMs, the presence of variable-length input sequences necessitates the use of padding tokens when batching sequences, leading to ineffectual computations. It is also well known that LMs over-fit to the small task-specific training datasets used during fine-tuning, despite the use of known regularization techniques. Based on these insights, we present TokenDrop + BucketSampler, a framework that synergistically combines a new regularizer that drops a random subset of insignificant words in each sequence in every epoch, and a length-aware batching method to simultaneously reduce padding and address the overfitting issue.</p> <p><br></p> <p>Next, we address the computational challenges of Transformers used for processing inputs of several important modalities, such as text, images, audio and videos. We present Input Compression with Positional Consistency (ICPC), a new data augmentation method that applies varying levels of compression to each training sample in every epoch, thereby simultaneously reducing over-fitting and improving training efficiency. ICPC also enables efficient variable-effort inference, where easy samples can be inferred at high compression levels, and vice-versa.</p> <p><br></p> <p>Finally, we focus on optimizing Graph Neural Networks (GNNs), which are commonly used for learning on non-Euclidean data. Few-shot learning with GNNs is an important challenge, since real-world graphical data is often sparsely labeled. Self-training, wherein the GNN is trained in stages by augmenting the training data with a subset of the unlabeled data and their pseudolabels, has emerged as a promising approach. However, self-training significantly increases the computational demands of training. We propose FASTRAIN-GNN, a framework for efficient and accurate self-training of GNNs with few labeled nodes. FASTRAIN-GNN optimizes the GNN architecture, training data, training parameters, and the graph topology during self-training.</p> <p><br></p> <p>At inference time, we find that ensemble GNNs are significantly more accurate and robust than single-model GNNs, but suffer from high latency and storage requirements. To address this challenge, we propose GNN Ensembles through Error Node Isolation (GEENI). The key concept in GEENI is to identify nodes that are likely to be incorrectly classified (error nodes) and suppress their outgoing messages, leading to simultaneous accuracy and efficiency improvements. </p> <p><br></p>

Recently, machine learning and deep neural networks (DNNs) have gained a significant amount of attention since they have achieved human-like performance in various tasks, such as image classification, recommendation, and natural language processing. As the tasks get more complicated, people build bigger and deeper networks to obtain high accuracy, and this brings challenges to existing hardware on fast and energy-efficient DNN computation due to the memory wall problem. First, traditional hardware spends a significant amount of energy on moving the data between memory and ALU units. Second, the traditional memory blocks only support row-by-row access, and this limits the computation speed and energy efficiency. In-memory computing (IMC) is one promising measure to solve the aforementioned problems in DNN computation. This approach combines the memory blocks with the computation units to enable high computation throughput and low energy consumption. On the macro level, both digital and analog-mixed-signal (AMS) IMC macros achieve high performance in the multiply-and-accumulation (MAC) computation. The AMS designs have high energy efficiency and highcompute density, and the digital designs have PVT robustness and technology scalability. On the architecture level, specialized hardware accelerators that integrate these IMC macros outperform the traditional hardware accelerators in end-to-end DNN inference. Beyond the IMC, other approaches also reduce energy consumption. For example, sparsity-aware training reduces the arithmetic energy by adding more zeros to the weights and zero-gating the multiplication and/or addition. Weight and activation compression reduces the off-chip memory access energy. This thesis presents new circuit and architecture designs for efficient DNN inference with in-memory computing architectures. First, this thesis presents two SRAM-based analog-mixed signal IMC macros. One is a macro with custom 10T1C cells for binary/ternary MAC operation. The other one, MACC-SRAM, is a multistep-accumulation capacitor-based IMC macro for 4b MAC computation. The macro features stepwise charging and discharging, sparsity optimization, and adder-first architecture for energy efficiency. Second, we propose a programmable DNN accelerator that integrates 108 AMS IMC macros. This accelerator, named PIMCA, with its own pipeline structure and instruction set architecture, can flexibly support the inference at the instruction level. Last, we implement a fully-digital accelerator that integrates IMC macros supporting floating-point number computation. The accelerator contains online decompression hardware to reducedata movement energy of weight and activation. It also contains online activation compressors to reduce the activation memory footprint.

The Cloud Radio Access Network (RAN) is a groundbreaking technology employed in the telecommunications industry, offering flexible, scalable, and cost-effective solutions for seamless wireless network services. However, testing Cloud RAN applications presents significant challenges due to their complexity, potentially leading to delays and increased costs. A paramount solution to overcome these obstacles is test automation. Automating the testing process not only dramatically reduces manual efforts but also enhances testing accuracy and efficiency, expediting the delivery of high-quality products. In the current era of cutting-edge advancements, artificial intelligence (AI) and machine learning (ML) play a transformative role in revolutionizing Cloud RAN testing. These innovative technologies enable rapid identification and resolution of complex issues, surpassing traditional methods. The objective of this thesis is to adopt an AI-enabled approach to Cloud RAN test automation, harnessing the potential of machine learning and natural language processing (NLP) techniques to automatically select test cases from test instructions. Through thorough analysis, relevant keywords are extracted from the test instructions using advanced NLP techniques. The performance of three keyword extraction methods is compared, with SpaCy proving to be the superior keyword extractor. Using the extracted keywords, test script prediction is conducted through two distinct approaches: using test script names and using test script contents. In both cases, Random Forest emerges as the top-performing model, showcasing its effectiveness with diverse datasets, regardless of oversampling or undersampling data augmentation techniques. Based on the rule-based approach, the predicted test scripts are utilized to determine the order of execution among the predicted test scripts. The research findings highlight the significant impact of AI and ML techniques in streamlining test case selection and automation for Cloud RAN applications. The proposed AI-enabled approach optimizes the testing process, resulting in faster product delivery, reduced manual workload, and overall cost savings.

Everyday we are flooded with news from all around the world and this information can be overwhelming. In our study we analyze the possibilities to implement GPT-3 models in the work of news summarization in swedish and automize this process. In this study we also regard the ethic point of view, meaning if we can trust these GPT-3 models and  give them the responsibility to make news summarizations. We studied three different GPT-3 models: ChatGPT, Megatron and GPT-SW3. We used a quantitative survey method where the participants got to rate the news summarizations made by the GPT-3 models. The participants got to rate the news summarizations based on the criterias language, contents and structure. We then took the mean value of the ratings from the survey to see the results. The results showed that ChatGPT was significantly the best of all the three GPT-models on all three criterias, and Megatron and GPT-SW3 performed significantly worse. This shows that these models still need some development to get to the same levels as ChatGPT. Despite ChatGPT being the best performing GPT-3 model it still had its weak sides. We noticed this through one article that had alot of factors included which meant alot of information for the GPT-3 models to condense. Through this study we could confirm that GPT-3 models who are further in their development, like ChatGPT can be used in the work of news summarization but should be used with cautioun of what articles it gets to summarize. This means that GPT-3 models still require human supervision for articles with too much information to condense.