"Secondary teachers across the United States are being asked to use formative assessment data to inform their classroom instruction. At the same time, critics of US government’s No Child Left Behind legislation are calling the bill “No Child Left Untested”. Among other things, critics point out that every hour spent assessing students is an hour lost from instruction. But, does it have to be? What if we better integrated assessment into classroom instruction and allowed students to learn during the test? This dissertation emphasizes using the intelligent tutoring system as an assessment system that just so happens to provide instructional assistance during the test. Usually it is believed that assessment get harder if students are allowed to learn during the test, as it’s then like trying to hit a moving target. So, my results are somewhat shocking that by providing tutoring to students while they are assessed I actually improve the assessment of students’ knowledge. Most traditional assessments treat all questions on the test as sampling a single underlying knowledge component. Yet, teachers want detailed, diagnostic reports to inform their instruction. Can we have our cake and eat it, too? In this dissertation, I provide solid evidence that a fine-grained skill model is able to predict state test scores better than coarser-rained models, as well as being used to give teachers more informative feedback that they can reflect on to improve their instruction. The contribution of the dissertation lies in that it established novel assessment methods to better assess students in intelligent tutoring systems. Through analyzing data of more than 1,000 students across two years, it provides strong evidence implying that it is possible to develop a continuous assessment system that can do all three of these things at the same time: 1) accurately and longitudinally assesses students, 2) gives fine grained feedback that is more cognitively diagnostic, and 3) saves classroom instruction time by assessing students while they are getting tutoring. "

"Intelligent Tutoring Systems have historically been shown to be an effective means of educating an audience. While there is great benefit from such systems they are generally very costly to build and maintain. It has been estimated that 200 hours of time is required to produce one hour of Intelligent Tutoring System content. The Office of Navel Research has funding this thesis because they are interested in reducing the cost of construction for Intelligent Tutoring Systems. In order for Intelligent Tutoring Systems to be widely accepted and used in the classroom environment there needs to be a toolset that allows for even the most novice user to maintain and grow the system with minimal cost. The goal of this thesis is to create such a toolset targeted towards the Assistments Project. One of the goals of the Assistments Project is to provide a means for teachers to receive meaningful data from the system that they can take to the classroom environment thus enabling a comprehensive learning solution. The effectiveness of the toolset was measured by its ability to reduce the overall time taken to package and distribute content in an Intelligent Tutoring System by providing the tools and allowing the completion of the tasks to be at a reasonable speed."

Two of the major goals in Educational Data Mining are determining students’ state of knowledge and determining their affective state. It is useful to be able to determine whether a student is engaged with a tutor or task in order to adapt to his/her needs and necessary to have an idea of the students' knowledge state in order to provide material that is appropriately challenging. These two problems are usually examined separately and multiple methods have been proposed to solve each of them. However, little work has been done on examining both of these states in parallel and the combined effect on a student’s performance. The work reported in this thesis explores ways to observe both behavior and performance in order to more fully understand student state.

Numerous methods to assess student knowledge are present throughout every step of a students€™ education. Skill-based assessments include homework, quizzes and tests while curriculum exams comprise of the SAT and GRE. The latter assessments provide an indication as to how well a student has retained a learned national curriculum however they are unable to identify how well a student performs at a fine grain skill level. The former assessments hone in on a specific skill or set of skills, however, they require an excessive amount of time to collect curriculum-wide data. We've developed a system that assesses students at a fine grain level in order to identify non- mastered skills within each student€™s zone of proximal development. €œPLACEments€� is a graph-driven computer adaptive test which not only provides thorough student feedback to educators but also delivers a personalized remediation plan to each student based on his or her identified non-mastered skills. As opposed to predicting state test scores, PLACEments objective is to personalize learning for students and encourage teachers to employ formative assessment techniques in the classroom. We have conducted a randomized controlled study to evaluate the learning value PLACEments provides in comparison to traditional methods of targeted skill mastery and retention.

Over the last two decades, the field of educational data mining (EDM) has been focusing on predicting the correctness of the next student response to the question (e.g., [2, 6] and the 2010 KDD Cup), in other words, predicting student short-term performance. Student modeling has been widely used for making such inferences. Although performing well on the immediate next problem is an indicator of mastery, it is by far not the only criteria. For example, the Pittsburgh Science of Learning Center's theoretic framework focuses on robust learning (e.g., [7, 10]), which includes the ability to transfer knowledge to new contexts, preparation for future learning of related skills, and retention - the ability of students to remember the knowledge they learned over a long time period. Especially for a cumulative subject such as mathematics, robust learning, particularly retention, is more important than short-term indicators of mastery. The Automatic Reassessment and Relearning System (ARRS) is a platform we developed and deployed on September 1st, 2012, which is mainly used by middle-school math teachers and their students. This system can help students better retain knowledge through automatically assigning tests to students, giving students opportunity to relearn the skill when necessary and generating reports to teachers. After we deployed and tested the system for about seven months, we have collected 287,424 data points from 6,292 students. We have created several models that predict students' retention performance using a variety of features, and discovered which were important for predicting correctness on a delayed test. We found that the strongest predictor of retention was a student's initial speed of mastering the content. The most striking finding was that students who struggled to master the content (took over 8 practice attempts) showed very poor retention, only 55% correct, after just one week. Our results will help us advance our understanding of learning and potentially improve ITS.

Schools use commercial systems specifically for mathematics benchmarking and longitudinal assessment. However these systems are expensive and their results often fail to indicate a clear path for teachers to differentiate instruction based on students’ individual strengths and weaknesses in specific skills. ASSISTments is a web-based Intelligent Tutoring System used by educators to drive real-time, formative assessment in their classrooms. The software is used primarily by mathematics teachers to deliver homework, classwork and exams to their students. We have developed a computer adaptive test called PLACEments as an extension of ASSISTments to allow teachers to perform individual student assessment and by extension school-wide benchmarking. PLACEments uses a form of graph-based knowledge representation by which the exam results identify the specific mathematics skills that each student lacks. The system additionally provides differentiated practice determined by the students’ performance on the adaptive test. In this project, we describe the design and implementation of PLACEments as a skill assessment method and evaluate it in comparison with a fixed-item benchmark.

L’adoption des Nouvelles Technologies de l’Information et de la Communication (NTIC) a permis la modernisation des méthodes d’enseignement dans les systèmes d’apprentissage en ligne comme l’e-Learning, les systèmes tutoriels intelligents, etc. Ces derniers assurent une formation à distance qui répond aux besoins des apprenants. Un aspect très important à prendre en considération dans ces systèmes est l’évaluation précoce de l’apprenant en termes d’acquisition des connaissances. En général, trois types d’évaluation et leurs relations sont nécessaires durant le processus d’apprentissage, à savoir : (i) diagnostic qui est exécuté avant l’apprentissage pour estimer le niveau des élèves, (ii) évaluation formative qui est appliquée lors de l’apprentissage pour tester l’évolution des connaissances et (iii) évaluation sommative qui est considérée après l’apprentissage pour évaluer l’acquisition des connaissances. Ces méthodes peuvent être intégrées d’une manière semi-automatique, automatique ou adaptée aux différents contextes de formation, par exemple dans le domaine d’apprentissage des langues (français, anglais, etc.), des sciences fondamentales (mathématiques, physique, chimie, etc.) et langages de programmation (java, python, sql, etc.) Cependant, les méthodes d’évaluation usuelles sont statiques et se basent sur des fonctions linéaires qui ne prennent en considération que la réponse de l’apprenant. Elles ignorent, en effet, d’autres paramètres de son modèle de connaissances qui peuvent divulguer d’autres indicateurs de performance. Par exemple, le temps de résolution d’un problème, le nombre de tentatives, la qualité de la réponse, etc. Ces éléments servent à détecter les traits du profil, le comportement ainsi que les troubles d’apprentissage de l’apprenant. Ces paramètres additionnels sont vus dans nos travaux de recherche comme des traces d’apprentissage produites par l’apprenant durant une situation ou un contexte pédagogique donné. Dans ce cadre, nous proposons dans cette thèse une approche d’évaluation de l’apprenant à base des traces d’apprentissage qui peut être exploitée dans un système d’adaptation de la ressource et/ou de la situation pédagogique. Pour l’évaluation de l’apprenant, nous avons proposé trois modèles génériques d’évaluation qui prennent en considération la trace temporelle, le nombre de tentatives et leurs combinaisons. Ces modèles ont servi, par la suite, comme métrique de base à notre modèle d’adaptation de la ressource et/ou de la situation d’apprentissage. Le modèle d’adaptation est également fondé sur les trois traces susmentionnées et sur nos modèles d’évaluation. Notre modèle d’adaptation génère automatiquement des trajectoires d’apprentissage adaptées en utilisant un modèle d’état-transition. Les états présentent des situations d’apprentissage qui consomment des ressources et les transitions entre situations expriment les conditions nécessaires à remplir pour passer d’une situation à une autre. Ces concepts sont aussi implémentés dans une ontologie du domaine et un algorithme d’adaptation a été également proposé. L’algorithme assure deux types d’adaptation : (i) Adaptation de la situation et (ii) Adaptation des ressources dans une situation. Afin de collecter les traces d’apprentissage pour la mise en œuvre de notre approche d’évaluation de l’apprenant et d’adaptation de ressources et de situations d’apprentissage, nous avons effectué des expérimentations sur deux groupes d’étudiants en Licence Informatique (L2). Un groupe en apprentissage classique et un groupe en apprentissage adapté. Sur la base des traces obtenues des sessions de travail des étudiants, nous avons utilisé nos modèles d’évaluation dont les résultats ont été utilisés pour mettre en œuvre l’adaptation. Après comparaison des résultats de l’apprentissage adapté à ceux obtenus de l’apprentissage classique, nous avons constaté une amélioration des résultats en termes de moyenne générale et d’écart-type des moyennes des apprenants. / The adoption of new Information and Communication Technologies (ICT) has enabled the modernization of teaching methods in online learning systems such as e-Learning, intelligent tutorial systems (ITS), etc. These systems provide a remote training that which meets the learner needs. A very important aspect to consider in these systems is the early assessment of the learner in terms of knowledge acquisition. In general, three types of assessment and their relationships are needed during the learning process, namely : (i) diagnostic which is performed before learning to estimate the level of students, (ii) formative evaluation which is applied during learning to test the knowledge evolution and (iii) summative evaluation which is considered after learning to evaluate learner’s knowledge acquisition. These methods can be integrated into a semi-automatic, automatic or adapted way in different contexts of formation, for example in the field of languages literary learning such as French, English, etc., hard sciences (mathematics, physics, chemistry, etc.) and programming languages (java, python, sql, etc.). However, the usual evaluation methods are static and are based on linear functions that only take into account the learner’s response. They ignore other parameters of their knowledge model that may disclose other performance indicators. For example, the time to solve a problem, the number of attempts, the quality of the response, etc. These elements are used to detect the profile characteristics, behavior and learning disabilitiesof the learner. These additional parameters are seen in our research as learning traces produced by the learner during a given situation or pedagogical context. In this context, we propose in this thesis a learner evaluation approach based on learning traces that can be exploited in an adaptation system of the resource and/or the pedagogic situation. For the learner assessment, we have proposed three generic evaluation models that take into consideration the temporal trace, number of attempts and their combinations. These models are later used as a base metric for our resource adaptation model and/or learning situation. The adaptation model is also based on the three traces mentioned above and on our evaluation models. Our adaptation model automatically generates adapted paths using a state-transition model. The states represent learning situations that consume resources and the transitions between situations express the necessary conditions to pass from one situation to another. These concepts are implemented in a domain ontology and an algorithm that we have developed. The algorithm ensures two types of adaptation : (i) Adaptation of the situation and (ii) Adaptation of resources within a situation. In order to collect traces of training for the implementation of our approaches of learner evaluation and adaptation of resources and learning situations, we conducted experiments on two groups of students in Computer Science (L2). One group in classical training and the other group in adapted training. Based on the obtained traces from the students’ training sessions, we assessed merners based on our evaluation models. The results are then used to implement the adaptation in a domain ontology. The latter is implemented within oracle 11g which allows a rule-based semantic reasoning. After comparing the results of the adapted training with those obtained from the classical one, we found an improvement in the results in terms of general average and standard deviation of the learner averages.

By opening the learner model to both the learner and other peers within an e-learning system, the learner gains control over his or her learner model and is able to reflect on the contents presented in the model. Many current modeling systems translate an existing model to fit the context when information is needed. This thesis explores the observation that information in the model depends on the context in which it is generated and describes a method of generating the model for the specific user and purpose. The main advantage of this approach is that exactly the right information is generated to suit the context and needs of the learner. To explore the benefits and possible downsides of this approach, a learner model Query Tool was implemented to give instructors and learners the opportunity to ask specific questions (queries) of the content delivery system hosting several online courses. Information is computed in real time when the query is run by the instructor, so the data is always up-to-date. Instructors may then choose to allow students to run the query as well, enabling learner reflection on their progress in the course as the instructor has defined it. I have called this process active open learner modelling, referring to the open learner modelling community where learner models are accessible by learners for reflective purposes, and referring to the active learner modelling community which describes learner modelling as a context-driven process. Specific research questions explored in this thesis include "how does context affect the modelling process when learner models are opened to users", "how can privacy be maintained while useful information is provided", and "can an accurate and useful learner model be computed actively".

For many courses, mathematics included, there is an associated interactive e-learning system that provides assessment and tutoring. Some of these systems are classified as Intelligent Tutoring Systems. MyMathLab, Mathzone, and Assessment of LEarning in Knowledge Space (ALEKS) are just a few of the interactive e-learning systems in mathematics. In ALEKS, assessment and tutoring are based on the Knowledge Space Theory. Previous studies in a traditional learning environment have shown ALEKS users to perform equally or better in mathematics achievement than the group who did not use ALEKS. The purpose of this research was to investigate the effect of ALEKS on students’ achievement in mathematics in an online learning environment and to determine the cognitive complexity of mathematical tasks enacted by ALEKS’s initial (pretest) and final (posttest) assessments. The targeted population for this study was undergraduate students in College Mathematics I, in an online course at a private university in the southwestern United States. The study used a quasi-experimental One-Group non-randomized pretest and posttest design. Five methods of analysis and one model were used in analyzing data: t-test, correctional analysis, simple and multiple regression analysis, Cronbach’s Alpha reliability test and Webb’s depth of knowledge model. A t-test showed a difference between the pretest and posttest reports, meaning ALEKS had a significant effect on students’ mathematics achievement. The correlation analysis showed a significant positive linear relationship between the concept mastery reports and the formative and summative assessments reports meaning there is a direct relationship between the ALEKS concept mastery and the assessments. The regression equation showed a better model for predicting mathematics achievement with ALEKS when the time spent learning in ALEKS and the concept mastery scores are used as part of the model. According to Webb’s depth of knowledge model, the cognitive complexity of the pretest and posttest question items used by ALEKS were as follows: 50.5% required application of skills and concepts, 37.1% required recall of information, and 12.4% required strategic thinking: None of the questions items required extended thinking or complex reasoning, implying ALEKS is appropriate for skills and concepts building at this level of mathematics.

Emotions play a significant role in many human mental activities, including decision-making, motivation, and cognition. Various intelligent and expert systems can be empowered with emotionally intelligent capabilities, especially systems that interact with humans and mimic human behaviour. However, most current methods in affect recognition studies use intrusive, lab-based, and expensive tools which are unsuitable for real-world situations. Inspired by studies on keystrokes dynamics, this thesis investigates the effectiveness of diagnosing users’ affect through their typing behaviour in an educational context. To collect users’ typing patterns, a field study was conducted in which subjects used a dialogue-based tutoring system built by the researcher. Eighteen dialogue features associated with subjective and objective ratings for users’ emotions were collected. Several classification techniques were assessed in diagnosing users’ affect, including discrimination analysis, Bayesian analysis, decision trees, and neural networks. An artificial neural network approach was ultimately chosen as it yielded the highest accuracy compared with the other methods. To lower the error rate, a hierarchical classification was implemented to first classify user emotions based on their valence (positive or negative) and then perform a finer classification step to determining which emotions the user experienced (delighted, neutral, confused, bored, and frustrated). The hierarchical classifier was successfully able to diagnose users' emotional valence, while it was moderately able to classify users’ emotional states. The overall accuracy obtained from the hierarchical classifier significantly outperformed previous dialogue-based approaches and in line with some affective computing methods.