The educational industry has evolved with the development of computer technology. The online education system (OES) provides a more effective and efficient educational strategy for students benefiting from computer science technologies. There is a need for a mapping of knowledge definition from the traditional education system to the data gathered from the non-traditional OES. This would make the measurement of knowledge possible for OES. The study aims to (a) find an appropriate knowledge definition through a literature review process, and (b) based on the definition measure students’ knowledge in OES, such as Hypocampus, by using machine learning techniques. Experiments were conducted using a well-known Bayesian KnowledgeTracing (BKT) model. The evaluation was performed on 3300 students studying medicine in France using Hypocampus OES. As a result, the student’s knowledge was measured in skills, performance, and achievement per subject with 81% average accuracy. The obtained results suggest the potential of the presented approach for measuring students’ knowledge in OES.

The goal of Intelligent Tutoring systems (ITSs) is to engage the students in sustained reasoning activity and to interact with students based on a deep understanding of student behavior. In order to understand student behavior, ITSs rely on student modeling methods to observes student actions in the tutor and creates a quantitative representation of student knowledge, interests, affective states. Good student models are going to effectively help ITSs customize instructions, engage student's interest and then promote learning. Thus, the work of building ITSs and advancing student modeling should be considered as two interconnected components of one system rather than two separate topics. In this work, we utilized the theoretical support of a well-known learning science theory, the spacing effect, to guide the development of an ITS, called Automatic Reassessment and Relearning System (ARRS). ARRS not only validated the effectiveness of spacing effect, but it also served as a testing field which allowed us to find out new approaches to improve student learning by conducting large-scale randomized controlled trials (RCTs). The rich data set we gathered from ARRS has advanced our understanding of robust learning and helped us build student models with advanced data mining methods. At the end, we designed a set of API that supports the development of ARRS in next generation ASSISTments platform and adopted deep learning algorithms to further improve retention performance prediction. We believe our work is a successful example of combining theory and practice to advance science and address real- world problems.

In this dissertation, several approaches I have taken to build upon the student learning model are described. There are two focuses of this dissertation. The first focus is on improving the accuracy with which future student knowledge and performance can be predicted by individualizing the model to each student. The second focus is to predict how different educational content and tutorial strategies will influence student learning. The two focuses are complimentary but are approached from slightly different directions. I have found that Bayesian Networks, based on belief propagation, are strong at achieving the goals of both focuses. In prediction, they excel at capturing the temporal nature of data produced where student knowledge is changing over time. This concept of state change over time is very difficult to capture with classical machine learning approaches. Interpretability is also hard to come by with classical machine learning approaches; however, it is one of the strengths of Bayesian models and aids in studying the direct influence of various factors on learning. The domain in which these models are being studied is the domain of computer tutoring systems, software which uses artificial intelligence to enhance computer based tutorial instruction. These systems are growing in relevance. At their best they have been shown to achieve the same educational gain as one on one human interaction. Computer tutors have also received the attention of White House, which mentioned an tutoring platform called ASSISTments in its National Educational Technology Plan. With the fast paced adoption of these data driven systems it is important to learn how to improve the educational effectiveness of these systems by making sense of the data that is being generated from them. The studies in this proposal use data from these educational systems which primarily teach topics of Geometry and Algebra but can be applied to any domain with clearly defined sub-skills and dichotomous student response data. One of the intended impacts of this work is for these knowledge modeling contributions to facilitate the move towards computer adaptive learning in much the same way that Item Response Theory models facilitated the move towards computer adaptive testing.

Transformer models, delivering big improvement in AI text-models (NLP), are now being applied in Knowledge Tracing to track the knowledge of students over time. One of the first, SAINT, showed quite some improvement over the then SOTA results on the public EdNet dataset and caused an increase in research based on transformer-based models. In this paper, we firstly aim to reproduce the SAINT results on the EdNet dataset but are unable to report a similar performance as the original paper. This might be due to implementation details, which we were not able to completely reconstruct. We hope to pave the road for further reproducibility, as an increasingly important part of AI research. Furthermore, we apply the model to a company dataset much larger than any public dataset (more interactions, more exercises and more skills). Such a dataset is on the one hand more challenging (more skills mixed), and on the other hand, provides much more data (which should help our models). We compare the SAINT model and the seminal IRT model, and find that the SAINT model performance is 4% better in AUC but 1.7% worse in RMSE. Our experiments on window size suggest that transformer models still struggle with modelling beyond recent performance, and do not yet deliver the step-change observed in NLP. / Transformermodeller, som ger stora förbättringar av AI-textmodeller (NLP), används nu i Knowledge Tracing för att spåra elevernas kunskaper över tid. En av de första, SAINT, visade en hel del förbättring jämfört med de dåvarande SOTA-resultaten på den offentliga EdNet-datauppsättningen och orsakade en ökning av forskning baserad på transformerbaserade modeller. I denna artikeln siktar vi först efter att återskapa SAINT-resultaten på EdNet-datauppsättningen, men vi kan inte rapportera liknande prestanda som den ursprungliga uppsatsen. Detta kan bero på implementeringsdetaljer som vi inte kunde rekonstruera helt. Vi hoppas kunna bana väg för ytterligare reproduktioner, som en allt viktigare del av AI-forskningen. Dessutom tillämpar vi modellen på en företagsdatauppsättning som är mycket större än någon offentlig datauppsättning (fler interaktioner, fler övningar och fler färdigheter). En sådan datauppsättning är å ena sidan mer utmanande (mer blandad kompetens), men å andra sidan ger den mycket mer data (vilket borde hjälpa våra modeller). Vi jämför SAINT-modellen och den framträdande IRT-modellen och finner att SAINT-modellens prestanda är 4% bättre i AUC men 1,7% sämre i RMSE. Våra experiment på fönsterstorlek tyder på att transformermodeller fortfarande kämpar med modellering utöver de senaste prestanda och ännu inte levererar den stegförändring som observerats i NLP.

Knowledge Tracing aims to predict future performance of users of learning platforms based on historical data, by modeling their knowledge state. In this task, the target is a binary variable representing the correctness of the exercise, where an exercise is a word uttered by the user. Current state-of-the-art models add attention layers to autoregressive models or rely on self-attention networks. However, these models are built on publicly available datasets that lack useful information about the interactions users have with exercises. In this work, various techniques are introduced that allow for the incorporation of additional information made available in a dataset provided by Astrid Education. They consist of encoding a time dimension, modeling the skill needed for each exercise explicitly, and adjusting the length of the interaction sequence. Introducing new information to the Knowledge Tracing framework allows Astrid to craft a more personalized experience for its users; thus fulfilling the purpose and goal of the thesis. Additionally, we perform experiments to understand what aspects influence the models. Results show that modeling the skills needed to solve an exercise using an encoding strategy and reducing the length of the interaction sequence lead to improvements in terms of both accuracy and AUC. The time-encoding did not lead to better results, further experimentation is needed to include the time dimension successfully. / Mänsklig kunskap är ett försök att förutsäga användarnas framtida prestanda på lärandeplattformar baserat på historiska data, genom att modellera deras kunskaps tillstånd. I denna uppgift är målet en binär variabel som representerar överensstämmelsen av övningen. Nuvarande state-of-the-art-modeller lägger till uppmärksamhetslager på autoregressiva modeller eller förlitar sig på self-attention-nätverk. Dessa modeller bygger dock på offentligt tillgängliga databaser som saknar användbar information om de interaktioner som användare har med övningar. I detta arbete introduceras olika tekniker som gör det möjligt att inkludera ytterligare information som görs tillgänglig i en databas som tillhandahålls av Astrid Education AB. De består av att koda en tidsdimension, modellera färdigheten som krävs för varje övning explicit och justera interaktionssekvenslängden. Genom att introducera ny information i ramverket för kunskapstracing tillåter Astrid att skapa en mer personlig upplevelse för sina användare; därmed uppfyller syftet och målet med denna avhandling. Dessutom genomför vi experiment för att förstå vilka aspekter som påverkar modellerna. Resultaten visar att modellering av färdigheter med en kodningsstrategi och reducering av interaktionssekvenslängden leder till förbättringar både vad gäller noggrannhet och AUC. Tidskodningen ledde inte till bättre resultat, ytterligare experimentering krävs för att inkludera tidsdimensionen på ett framgångsrikt sätt.

Despite widespread recognition by science educators, researchers and K-12 frameworks that scientific inquiry should be an essential part of science education, typical classrooms and assessments still emphasize rote vocabulary, facts, and formulas. One of several reasons for this is that the rigorous assessment of complex inquiry skills is still in its infancy. Though progress has been made, there are still many challenges that hinder inquiry from being assessed in a meaningful, scalable, reliable and timely manner. To address some of these challenges and to realize the possibility of formative assessment of inquiry, we describe a novel approach for evaluating, tracking, and scaffolding inquiry process skills. These skills are demonstrated as students experiment with computer-based simulations. In this work, we focus on two skills related to data collection, designing controlled experiments and testing stated hypotheses. Central to this approach is the use and extension of techniques developed in the Intelligent Tutoring Systems and Educational Data Mining communities to handle the variety of ways in which students can demonstrate skills. To evaluate students' skills, we iteratively developed data-mined models (detectors) that can discern when students test their articulated hypotheses and design controlled experiments. To aggregate and track students' developing latent skill across activities, we use and extend the Bayesian Knowledge-Tracing framework (Corbett & Anderson, 1995). As part of this work, we directly address the scalability and reliability of these models' predictions because we tested how well they predict for student data not used to build them. When doing so, we found that these models demonstrate the potential to scale because they can correctly evaluate and track students' inquiry skills. The ability to evaluate students' inquiry also enables the system to provide automated, individualized feedback to students as they experiment. As part of this work, we also describe an approach to provide such scaffolding to students. We also tested the efficacy of these scaffolds by conducting a study to determine how scaffolding impacts acquisition and transfer of skill across science topics. When doing so, we found that students who received scaffolding versus students who did not were better able to acquire skills in the topic in which they practiced, and also transfer skills to a second topic when was scaffolding removed. Our overall findings suggest that computer-based simulations augmented with real-time feedback can be used to reliably measure the inquiry skills of interest and can help students learn how to demonstrate these skills. As such, our assessment approach and system as a whole shows promise as a way to formatively assess students' inquiry.

Despite widespread recognition by science educators, researchers and K-12 frameworks that scientific inquiry should be an essential part of science education, typical classrooms and assessments still emphasize rote vocabulary, facts, and formulas. One of several reasons for this is that the rigorous assessment of complex inquiry skills is still in its infancy. Though progress has been made, there are still many challenges that hinder inquiry from being assessed in a meaningful, scalable, reliable and timely manner. To address some of these challenges and to realize the possibility of formative assessment of inquiry, we describe a novel approach for evaluating, tracking, and scaffolding inquiry process skills. These skills are demonstrated as students experiment with computer-based simulations. In this work, we focus on two skills related to data collection, designing controlled experiments and testing stated hypotheses. Central to this approach is the use and extension of techniques developed in the Intelligent Tutoring Systems and Educational Data Mining communities to handle the variety of ways in which students can demonstrate skills. To evaluate students' skills, we iteratively developed data-mined models (detectors) that can discern when students test their articulated hypotheses and design controlled experiments. To aggregate and track students' developing latent skill across activities, we use and extend the Bayesian Knowledge-Tracing framework (Corbett & Anderson, 1995). As part of this work, we directly address the scalability and reliability of these models' predictions because we tested how well they predict for student data not used to build them. When doing so, we found that these models demonstrate the potential to scale because they can correctly evaluate and track students' inquiry skills. The ability to evaluate students' inquiry also enables the system to provide automated, individualized feedback to students as they experiment. As part of this work, we also describe an approach to provide such scaffolding to students. We also tested the efficacy of these scaffolds by conducting a study to determine how scaffolding impacts acquisition and transfer of skill across science topics. When doing so, we found that students who received scaffolding versus students who did not were better able to acquire skills in the topic in which they practiced, and also transfer skills to a second topic when was scaffolding removed. Our overall findings suggest that computer-based simulations augmented with real-time feedback can be used to reliably measure the inquiry skills of interest and can help students learn how to demonstrate these skills. As such, our assessment approach and system as a whole shows promise as a way to formatively assess students' inquiry.

Knowledge tracing is concerned with tracking a student’s knowledge as she/he engages with exercises in an (online) learning platform. A commonly used state-of-theart knowledge tracing model is Deep Knowledge Tracing (DKT) which models the time dimension as a sequence of completed exercises per student by using a Long Short-Term Memory Neural Network (LSTM). However, a common problem in this sequence-based model is too much instability in the time dimension of the modelled knowledge of a student. In other words, the student’s knowledge on a skill changes too quickly and unreliably. We propose dynamic student embeddings as a stable method for encoding the time dimension of knowledge tracing systems. In this method the time dimension is encoded in time slices of a fixed size, while the model’s loss function is designed to smoothly align subsequent time slices. We compare the dynamic student embeddings to DKT on a large-scale real-world dataset, and we show that dynamic student embeddings provide a more stable knowledge tracing while retaining good performance. / Kunskapsspårning handlar om att modellera en students kunskaper då den arbetar med uppgifter i en (online) lärplattform. En vanlig state-of-the-art kunskapsspårningsmodell är Deep Knowledge Tracing (DKT) vilken modellerar tidsdimensionen som en sekvens av avslutade uppgifter per student med hjälp av ett neuronnät kallat Long Short-Term Memory Neural Network (LSTM). Ett vanligt problem i dessa sekvensbaserade modeller är emellertid en för stor instabilitet i tidsdimensionen för studentens modellerade kunskap. Med andra ord, studentens kunskaper förändras för snabbt och otillförlitligt. Vi föreslår därför Dynamiska Studentvektorer som en stabil metod för kodning av tidsdimensionen för kunskapsspårningssystem. I denna metod kodas tidsdimensionen i tidsskivor av fix storlek, medan modellens förlustfunktion är utformad för att smidigt justera efterföljande tidsskivor. I denna uppsats jämför vi de Dynamiska Studentvektorer med DKT i en storskalig verklighetsbaserad dataset, och visar att Dynamiska Studentvektorer tillhandahåller en stabilare kunskapsspårning samtidigt som prestandan bibehålls.