<p>The present dissertation investigated the impact of scales / scoring methods and prompt linguistic features on the meausrement quality of L2 English elicited imitation (EI). Scales / scoring methods are an important feature for the validity and reliabilty of L2 EI test, but less is known (Yan et al., 2016). Prompt linguistic features are also known to influence EI test quaity, particularly item difficulty, but item discrimination or corpus-based, fine-grained meausres have rarely been incorporated into examining the contribution of prompt linguistic features. The current study addressed the research needs, using item response theory (IRT) and random forest modeling.</p><p>Data consisted of 9,348 oral responses to forty-eight items, including EI prompts, item scores, and rater comments, which were collected from 779 examinees of an L2 English EI test at Purdue Universtiy. First, the study explored the current and alternative EI scales / scoring methods that measure grammatical / semantic accuracy, focusing on optimal IRT-based measurement qualities (RQ1 through RQ4 in Phase Ⅰ). Next, the project identified important prompt linguistic features that predict EI item difficulty and discrimination across different scales / scoring methods and proficiency, using multi-level modeling and random forest regression (RQ5 and RQ6 in Phase Ⅱ).</p><p>The main findings were (although not limited to): 1) collapsing exact repetition and paraphrase categories led to more optimal measurement (i.e., adequacy of item parameter values, category functioning, and model / item / person fit) (RQ1); there were fewer misfitting persons with lower proficiency and higher frequency of unexpected responses in the extreme categories (RQ2); the inconsistency of qualitatively distinguishing semantic errors and the wide range of grammatical accuracy in the minor error category contributed to misfit (RQ3); a quantity-based, 4-category ordinal scale outperformed quality-based or binary scales (RQ4); sentence length significantly explained item difficulty only, with small variance explained (RQ5); Corpus-based lexical measures and phrase-level syntactic complexity were important to predicting item difficulty, particularly for the higher ability level. The findings made implications for EI scale / item development in human and automatic scoring settings and L2 English proficiency development.</p>

Behavioral skills training (BST) is often used to train caregivers to implement various behavior management procedures; however, additional strategies are sometimes required to promote the generalization of skills from a contrived setting to the natural environment. Generalizing skills to the natural environment requires that the caregiver's behavior transfer from control of stimuli in the contrived setting to stimuli in the natural environment, and the skill continues to be performed with high levels of accuracy. The purpose of this study was to assess the extent to which caregivers generalized the use of social reinforcement, in the form of descriptive praise, from the contrived setting to the natural environment. When caregivers failed to respond to opportunities, a progressive prompt delay was used to bring caregivers' responding under the control of relevant client behavior; feedback was used to improve the accuracy with which caregivers implemented reinforcement. Five caregivers in a large residential facility participated in the study; single-opportunity probes were used to assess caregiver's identification of opportunities and accuracy in implementing reinforcement for two defined client behaviors, compliance and appropriate attention-getting behavior. Results of the study suggest that skills failed to generalize from the contrived setting to the natural environment. However, prompting was effective in training caregivers to identify opportunities to provide reinforcement, and feedback improved implementation of reinforcement.

<p>Empowering machines to understand our physical world should go beyond models with only natural language and models with only vision. Vision and language is a growing field of study that attempts to bridge the gap between natural language processing and computer vision communities by enabling models to learn visually grounded language. However, as an increasing number of pre-trained visual linguistic models focus on the alignment between visual regions and natural language, it is difficult to claim that these models capture certain properties of objects in their latent space, such as size. Inspired by recent trends in prompt learning, this study will design a prompt learning framework for two visual linguistic models, ViLBERT and ViLT, and use different manually crafted prompt templates to evaluate the consistency of performance of these models in comparing the size of objects. The results of this study showed that ViLT is more consistent in prediction accuracy for the given task with six pairs of objects under four prompt designs. However, the overall prediction accuracy is lower than the expectation on this object size comparison task; even the better model in this study, ViLT, has only 16 out of 24 cases better than the proposed random chance baseline. As this study is a preliminary study to explore the potential of pre-trained visual linguistic models on object size comparison, there are many directions for future work, such as investigating more models, choosing more object pairs, and trying different methods for feature engineering and prompt engineering.</p>

The knowledge of tests conducted on a problematic vehicle is essential for enhancing the efficiency of mechanics. Therefore, identifying the tests performed in each repair case is of utmost importance. This thesis explores techniques for extracting data from unstructured repair reports to identify component tests. The main emphasis is on developing a supervised multi-class classifier to categorize data and extract sentences that describe repair diagnoses and actions. It has been shown that incorporating a category-aware contrastive learning objective can improve the repair report classifier’s performance. The proposed approach involves training a sentence representation model based on a pre-trained model using a category-aware contrastive learning objective. Subsequently, the sentence representation model is further trained on the classification task using a loss function that combines the cross-entropy and supervised contrastive learning losses. By applying this method, the macro F1-score on the test set is increased from 90.45 to 90.73. The attempt to enhance the performance of the repair report classifier using a noisy data classifier proves unsuccessful. The noisy data classifier is trained using a prompt-based fine-tuning method, incorporating open-ended questions and two examples in the prompt. This approach achieves an F1-score of 91.09 and the resulting repair report classification datasets are found easier to classify. However, they do not contribute to an improvement in the repair report classifier’s performance. Ultimately, the repair report classifier is utilized to aid in creating the input necessary for identifying component tests. An information retrieval method is used to conduct the test identification. The incorporation of this classifier and the existing labels when creating queries leads to an improvement in the mean average precision at the top 3, 5, and 10 positions by 0.62, 0.81, and 0.35, respectively, although with a slight decrease of 0.14 at the top 1 position.

As advancements in chatbots and Large Language Models (LLMs) such as GPT-3.5 and GPT-4 continue, their applications in diverse fields, including psychology, expand. This study investigates the effectiveness of LLMs optimized through prompt engineering, aiming to enhance their performance in psychological applications. To this end, two distinct versions of a GPT-3.5-based chatbot were developed: a version similar to the base model, and a version equipped with a more extensive system prompt detailing expected behavior. A panel of professional psychologists evaluated these models based on a predetermined set of questions, providing insight into their potential future use as psychological tools. Our results indicate that an overly prescriptive system prompt can unintentionally limit the versatility of the chatbot, making a careful balance in instruction specificity essential. Furthermore, while our study suggests that current LLMs such as GPT-3.5 are not capable of fully replacing human psychologists, they can provide valuable assistance in tasks such as basic question answering, consolation and validation, and triage. These findings provide a foundation for future research into the effective integration of LLMs in psychology and contribute valuable insights into the promising field of AI-assisted psychological services. / I takt med att framstegen inom chatbots och stora språkmodeller (LLMs) som GPT-3.5 och GPT-4 fortsätter utvidgas deras potentiella tillämpningar inom olika områden, inklusive psykologi. Denna studie undersöker effektiviteten av LLMs optimerade genom prompt engineering, med målet att förbättra deras prestanda inom psykologiska tillämpningar. I detta syfte utvecklades två distinkta versioner av en chatbot baserad på GPT-3.5: en version som liknar bas-modellen, och en version utrustad med en mer omfattande systemprompt som detaljerar förväntat beteende. En panel av professionella psykologer utvärderade dessa modeller baserat på en förbestämd uppsättning frågor, vilket ger inblick i deras potentiella framtida användning som psykologiska verktyg. Våra resultat tyder på att en överdrivet beskrivande systemprompt kan ofrivilligt begränsa chatbotens mångsidighet, vilket kräver en noggrann balans i specificiteten av prompten. Vidare antyder vår studie att nuvarande LLMs som GPT-3.5 inte kan ersätta mänskliga psykologer helt och hållet, men att de kan ge värdefull hjälp i uppgifter som grundläggande frågebesvaring, tröst och bekräftelse, samt triage. Dessa resultat ger en grund för framtida forskning om effektiv integration av LLMs inom psykologi och bidrar med värdefulla insikter till det lovande fältet av AI-assisterade psykologtjänster.

This graduation report explores the integration of Artificial Intelligence (AI) tools, specifically OpenAI's Generative Pre-trained Transformer (GPT) technology, into web development processes using WordPress (WP) for developing a AI-driven chat service. The focus of the project is on ImagineX AB, a private company that offers the educational service ChatGPT Utbildning aimed at teaching professionals to effectively utilize ChatGPT. The project is motivated by the rapid growth and adoption of AI tools such as ChatGPT, underpinned by the observed increase in user base and its integration into significant platforms, like Microsoft's Bing and Office packages. Despite its promising potential, the application of such AI tools in web development remains underexplored and untested in several aspects. The graduation report presents the implementation of a GPT model-driven chat service on the ChatGPT Utbildning WP website, enabling visitors to interact with the famous AI tool directly. This feature serves a dual purpose – enhancing user engagement and providing an instant demonstration of the utility of ChatGPT. The agile project management methodology in general is divided into four phases: preliminary work, design solutions, develop solution, and delivery – design and development phases are iterative. In this project, there is two design iterations and three development iterations called “cycles”. The project plan is fulfilled with no deviation. Tests and continuous improvements are done throughout the development, with specific and planned in each phase and cycle. The result is two optimized chat bots in respective well-designed chat boxes with full chat functionality driven by OpenAI API and GPT-3.5/GPT-4 models – user tested and then published on ChatGPT Utbildning website. Additionally, insights in agile management solutions in relation to AI tools have been produced. The detailed construction and in-depth discussion contribute to the wide understanding of AI implementation in web development, providing practical insights into the application of ChatGPT in a real-world setting by agile project management. Furthermore, it underscores the transformative potential of AI tools in shaping web solutions and web development, and propelling innovation in the field. The report delves into discussion of technology, ethics, society, and implications on future web development. / Rapporten ämnar redogöra integreringen av artificiell intelligens (AI) instrument, särskilt OpenAI's Generative Pre-trained Transformer (GPT) teknologi, inom ramen för webbutvecklingsprocesser, inklusive agil projektledning, med användning av WordPress (WP), i syfte att utveckla en AIdrivande chatttjänst. Fokus för projektet är på företaget ImagineX AB, en privat aktör som erbjuder en utbildningstjänst benämnd ChatGPT Utbildning med mål att undervisa yrkesverksamma i effektivt bruk av ChatGPT. Motivationen för projektet härstammar från den snabbt växande tillväxten och adoptionen av AI-instrument som ChatGPT, vilket stärks av den observerade tillväxten av användarbasen och dess integrering i betydande plattformar, såsom Microsofts Bing och Office-paket. Trots den lovande potential som dessa AIinstrument innehar, finns det fortfarande delar inom webbutveckling där användningen av sådana verktyg förblir ouppklarade och otillräckligt utforskade. Rapporten visar implementeringen av en GPT-modell-drivande chattjänst på ChatGPT Utbildning WP-webbplatsen, vilket möjliggör direkt interaktion för besökare med det framstående AI-instrumentet. Denna funktion har ett tvåfaldigt ändamål - att förhöja användarengagemang och att ge en omedelbar demonstration av ChatGPT:s användbarhet. Den använda smidiga projektledningsmetodiken är typiskt uppdelad i fyra faser: preliminärt arbete, designlösningar, utvecklingslösningar samt leverans - designoch utvecklingsfaser är iterativa vilket omfattar två designiterationer och tre utvecklingsiterationer refererade till som "cykler". Projektplanen har följts utan avvikelser. Testning och kontinuerliga förbättringar har genomförts under hela utvecklingsprocessen, med specifika och planerade insatser i varje fas och cykel. Resultatet manifesteras i två optimerade chattrobotar inom respektive välutformade chattfönster, med fullständig chattfunktionalitet som drivs av OpenAI API samt GPT-3.5/GPT-4 modellerna - vilka har användartestats och därefter publicerats på ChatGPT Utbildning webbplatsen. Ytterligare insikter rörande agil projektledning i relation till AI-frågor erhålls också. Den detaljerade konstruktionen och den djupgående diskussionen bidrar till en omfattande förståelse för AI-implementering inom webbutveckling och ger praktiska insikter om tillämpningen av ChatGPT i en realistisk inställning med smidig projektledning. Vidare framhäver det den transformerande potentialen hos AI-instrument för att utforma webblösningar och webbutveckling, vilket främjar innovation inom området. Rapporten avslutas med diskussioner kring teknik, etik, samhälle och implikationer för framtida webbutveckling.

From the minute a student walks into her first day of kindergarten, she is learning to read and write. Reading and writing are reciprocal in nature, using the same composing processes (Roe, Smith, & Kolodziej, 2019). Interchangeable thinking skills are essential for both reading and writing, such as analyzing, identifying, inferencing, evaluating, and comparing (Roe, Smith, & Koldziej, 2019). Published research over time suggested that instruction focused on teaching students the craft and mechanics of writing significantly contributed to the overall improvement across the spectrum of literacy development (Cutler, 2015;Raphael, 2019; Wright, 2016). However, studies also suggested that teachers of all grade levels tend to vary in their approach to teaching writing (Newmark, B., Speck, D., Amesbury, E., Lough, C., Belgutay, J., Lowe, J., … Hepburn, H, 2018). This study was focused on understanding how two elementary level teachers interpreted writing curriculum and carried out instruction in their respective classrooms. Qualitative methodological procedures were employed through interviewing both educators and observing their writing instruction. The collected data was analyzed through inductive thematic analysis and findings included: 1) both teachers believed that writing instruction matters; 2) both teachers followed the curriculum as they learned in teacher professional development; 3) writing instruction varied according to primary versus elementary contexts.

When reading a text, it is common to get stuck on unfamiliar words that are difficult to understand in the local context. In these cases, we use dictionaries or similar online resources to find the general meaning of the word. However, maintaining a handwritten dictionary is highly resource demanding as the language is constantly developing, and using generative language models for producing definitions could therefore be a more efficient option. To explore this possibility, this thesis performs an online survey to examine if GPT could be useful for defining words. It also investigates how well the Swedish language model GPT-SW3 (3.5 b) define words compared to the model text-davinci-003, and how prompts should be formatted when defining words with these models. The results indicate that text-davinci-003 generates high quality definitions, and according to students t-test, the definitions received significantly higher ratings from participants than definitions taken from Svensk ordbok (SO). Furthermore, the results showed that GPT-SW3 (3.5 b) received the lowest ratings, indicating that it takes more investment to keep up with the big models developed by OpenAI. Regarding prompt formatting, the most appropriate prompt format for defining words is highly dependent on the model, and the results showed that text- davinci-003 performed well using zero-shot, while GPT-SW3 (3.5 b) required a few shot setting. Considering both the high quality of the definitions generated by text-davinci-003, and the practical advantages with generating definitions automatically, GPT could be a useful method for defining words.

Question generation (QG), which automatically generates good-quality questions from a piece of text, is capable of lowering the cost of the manual composition of questions. Recently Question generation has attracted increasing interest for its ability to supply a large number of questions for developing conversation systems and educational applications, as well as corpus development for natural language processing (NLP) research tasks, such as question answering and reading comprehension. Previous neural-based QG approaches have achieved remarkable performance. In contrast, these approaches require a large amount of data to train neural models properly, limiting the application of question generation in low-resource scenarios, e.g. with a few hundred training examples. This thesis aims to address the problem of the low-resource scenario by investigating a recently emerged paradigm of NLP modelling, prompt-based learning. Prompt-based learning, which makes predictions based on the knowledge of the pre-trained language model and some simple textual task descriptions, has shown great effectiveness in various NLP tasks in few-shot and zero-shot settings, in which a few or non-examples are needed to train a model. In this project, we have introduced a prompt-based question generation approach by constructing question generation task instructions that are understandable by a pre-trained sequence-to-sequence language model. Our experiment results show that our approach outperforms previous state-of-the-art question generation models with a vast margin of 36.8%, 204.8%, 455.9%, 1083.3%, 57.9% for metrics BLEU-1, BLEU-2, BLEU-3, BLEU-4, and ROUGE-L respectively in the few-shot learning settings. We also conducted a quality analysis of the generated questions and found that our approach can generate questions with correct grammar and relevant topical information when training with as few as 1,000 training examples.

Making information stored in databases more accessible to users inexperienced in structured query language (SQL) by converting natural language to SQL queries has long been a prominent research area in both the database and natural language processing (NLP) communities. There have been numerous approaches proposed for this task, such as encoder-decoder frameworks, semantic grammars, and more recently with the use of large language models (LLMs). When training LLMs to successfully generate SQL queries from natural language questions there are three notable methods used, pretraining, transfer learning and in-context learning (ICL). ICL is particularly advantageous in scenarios where the hardware at hand is limited, time is of concern and large amounts of task specific labled data is nonexistent. This study seeks to evaluate two strategies in ICL, namely zero-shot and few-shot scenarios using the Mistral-7B-Instruct LLM. Evaluation of the few-shot scenarios was conducted using two techniques, random selection and Jaccard Similarity. The zero-shot scenarios served as a baseline for the few-shot scenarios to overcome, which ended as anticipated, with the few-shot scenarios using Jaccard similarity outperforming the other two methods, followed by few-shot scenarios using random selection coming in at second best, and the zero-shot scenarios performing the worst. Evaluation results acquired based on execution accuracy and exact matching accuracy confirm that leveraging similarity in demonstrating examples when prompting the LLM will enhance the models knowledge about the database schema and table names which is used during the inference phase leadning to more accurately generated SQL queries than leveraging diversity in demonstrating examples.