Empirical studies have shown that learning from worked examples is an effective learning strategy. A worked example provides step-by-step explanations of how a problem is solved. Many studies have compared learning from examples to unsupported problem solving, and suggested presenting worked examples to students in the initial stages of learning, followed by problem solving once students have acquired enough knowledge. Recently, researchers have started comparing learning from examples to supported problem solving in Intelligent Tutoring Systems (ITSs). ITSs provide multiple levels of assistance to students, adaptive feedback being one of them. The goal of this research is to investigate using examples in constraint-based tutors by adding examples into SQL-Tutor. SQL-Tutor is a constraint-based tutor that teaches the Structured Query Language (SQL). Students with different prior knowledge benefit differently from studying examples; thus, another goal of the research is to propose an adaptive model that considers the student’s prior knowledge for providing worked examples. Evaluation of this research produced promising results. First, a fixed sequence of alternating examples and problems was compared with problems only and examples only. The result shows that alternating examples and problems is superior to the other two conditions. Then, a study was conducted, in which a fixed sequence of alternating worked examples and tutored problem solving is compared with a strategy that adapts the assistance level to students’ needs. The adaptive strategy determines the type of the task (a worked example, a faded example or a problem to be solved) based on how much assistance the student received in the previous problem. The results show that students in the adaptive condition learnt significantly more than their peers who were presented with the fixed sequence of worked examples and problem solving. The final study employed eye tracking and demonstrated that novices and advanced students study SQL examples differently. Such information can be used to provide proactive rather than reactive feedback messages to students’ actions.

Educators and educational researchers constantly strive to find effective instructional methods that meet the needs of struggling students. There is a well-established relationship between self-regulated learning and academic achievement. Therefore, a great deal of research has been conducted examining the effectiveness of interventions designed to develop self-regulated learning sub-processes including goal setting, help-seeking behavior, self-monitoring, and causal attributions. One particular sub-process that has gained significant attention is self-motivation beliefs, which includes goal orientation. Developing a growth mindset, or the belief that that intelligence is malleable, has been found to increase student learning. Intelligent tutoring systems have also been incorporated into K-12 education to help differentiate instruction and improve learning outcomes. There have been several empirical studies that have attempted to develop help-seeking behavior and growth mindset with interventions delivered by intelligent tutoring systems. Initially, the goal of this dissertation was to increase student learning by developing self-regulated learning through the use of an intelligent tutoring system. Preliminary attempts failed to modify student beliefs and behavior. As a result, a series of additional randomized controlled trials were conducted. This dissertation is a compilation of those studies, which attempted to leverage ASSISTments, an intelligent tutoring system, to improve student learning in mathematics. Each randomized controlled trial introduced an intervention, based on prior work, designed to address at least one aspect of self-regulated learning and measure the effect on learning. Most of the studies were unsuccessful in producing significant changes in either self-regulation or learning, failing to support the findings of prior research. Survey results suggest that students are reluctant to engage in certain self-regulated learning behaviors, like self-recording, because of the frustration caused when answering a question incorrectly. Based on the findings from these studies, recommendations for potential interventions and future research are discussed.

"Secondary teachers across the United States are being asked to use formative assessment data to inform their classroom instruction. At the same time, critics of US government’s No Child Left Behind legislation are calling the bill “No Child Left Untested”. Among other things, critics point out that every hour spent assessing students is an hour lost from instruction. But, does it have to be? What if we better integrated assessment into classroom instruction and allowed students to learn during the test? This dissertation emphasizes using the intelligent tutoring system as an assessment system that just so happens to provide instructional assistance during the test. Usually it is believed that assessment get harder if students are allowed to learn during the test, as it’s then like trying to hit a moving target. So, my results are somewhat shocking that by providing tutoring to students while they are assessed I actually improve the assessment of students’ knowledge. Most traditional assessments treat all questions on the test as sampling a single underlying knowledge component. Yet, teachers want detailed, diagnostic reports to inform their instruction. Can we have our cake and eat it, too? In this dissertation, I provide solid evidence that a fine-grained skill model is able to predict state test scores better than coarser-rained models, as well as being used to give teachers more informative feedback that they can reflect on to improve their instruction. The contribution of the dissertation lies in that it established novel assessment methods to better assess students in intelligent tutoring systems. Through analyzing data of more than 1,000 students across two years, it provides strong evidence implying that it is possible to develop a continuous assessment system that can do all three of these things at the same time: 1) accurately and longitudinally assesses students, 2) gives fine grained feedback that is more cognitively diagnostic, and 3) saves classroom instruction time by assessing students while they are getting tutoring. "

"In the United States, many students are doing poorly on new high-stakes standards-based tests that are required by the No Child Left Behind Act of 2002. Teachers are expected to cover more material to address all of the topics covered in standardized tests, and instructional time is more precious than ever. Educators want to know that the interventions that they are using in their classrooms are effective for students of varying abilities. Many educational technologies rely on tutored problem solving, which requires students to work through problems step-by-step while the system provides hints and feedback, to improve student learning. Intelligent tutoring researchers, education scientists and cognitive scientists are interested in knowing whether tutored problem solving is effective and for whom. Intelligent tutoring systems have the ability to adapt to individual students but need to know what types of feedback to present to individual students for the best and most efficient learning results. This dissertation presents an evaluation of the ASSISTment System, an intelligent tutoring system for the domain of middle school mathematics. In general, students were found to learn when engaging in tutored problem solving in the ASSISTment System. Students using the ASSISTment System also learned more when compared to paper-and-pencil problem-solving. This dissertation puts together a series of randomized controlled studies to build a comprehensive theory about when different types of tutoring feedback are more appropriate in an intelligent tutoring system. Data from these studies were used to analyze whether interactive tutored problem solving in an intelligent tutoring system is more effective than less interactive methods of allowing students to solve problems. This dissertation is novel in that it presents a theory that designers of intelligent tutoring systems could use to better adapt their software to the needs of students. One of the interesting results showed is that the effectiveness of tutored problem solving in an intelligent tutoring system is dependent on the math proficiency of the students. Students with low math proficiency learned more when they engaged in interactive tutoring sessions where they worked on one step at a time, and students with high math proficiency learned more when they were given the whole solution at once. More interactive methods of tutoring take more time versus less interactive methods. The data showed that it is worth the extra time it takes for students with low math proficiency. The main contribution of this dissertation is the development of a comprehensive theory of when educational technologies should use tutored problem solving to help students learn compared to other feedback mechanisms such as hints on demand, worked out solutions, worked examples and educational web pages. "

Two of the major goals in Educational Data Mining are determining students’ state of knowledge and determining their affective state. It is useful to be able to determine whether a student is engaged with a tutor or task in order to adapt to his/her needs and necessary to have an idea of the students' knowledge state in order to provide material that is appropriately challenging. These two problems are usually examined separately and multiple methods have been proposed to solve each of them. However, little work has been done on examining both of these states in parallel and the combined effect on a student’s performance. The work reported in this thesis explores ways to observe both behavior and performance in order to more fully understand student state.

Numerous methods to assess student knowledge are present throughout every step of a students€™ education. Skill-based assessments include homework, quizzes and tests while curriculum exams comprise of the SAT and GRE. The latter assessments provide an indication as to how well a student has retained a learned national curriculum however they are unable to identify how well a student performs at a fine grain skill level. The former assessments hone in on a specific skill or set of skills, however, they require an excessive amount of time to collect curriculum-wide data. We've developed a system that assesses students at a fine grain level in order to identify non- mastered skills within each student€™s zone of proximal development. €œPLACEments€� is a graph-driven computer adaptive test which not only provides thorough student feedback to educators but also delivers a personalized remediation plan to each student based on his or her identified non-mastered skills. As opposed to predicting state test scores, PLACEments objective is to personalize learning for students and encourage teachers to employ formative assessment techniques in the classroom. We have conducted a randomized controlled study to evaluate the learning value PLACEments provides in comparison to traditional methods of targeted skill mastery and retention.

Over the last two decades, the field of educational data mining (EDM) has been focusing on predicting the correctness of the next student response to the question (e.g., [2, 6] and the 2010 KDD Cup), in other words, predicting student short-term performance. Student modeling has been widely used for making such inferences. Although performing well on the immediate next problem is an indicator of mastery, it is by far not the only criteria. For example, the Pittsburgh Science of Learning Center's theoretic framework focuses on robust learning (e.g., [7, 10]), which includes the ability to transfer knowledge to new contexts, preparation for future learning of related skills, and retention - the ability of students to remember the knowledge they learned over a long time period. Especially for a cumulative subject such as mathematics, robust learning, particularly retention, is more important than short-term indicators of mastery. The Automatic Reassessment and Relearning System (ARRS) is a platform we developed and deployed on September 1st, 2012, which is mainly used by middle-school math teachers and their students. This system can help students better retain knowledge through automatically assigning tests to students, giving students opportunity to relearn the skill when necessary and generating reports to teachers. After we deployed and tested the system for about seven months, we have collected 287,424 data points from 6,292 students. We have created several models that predict students' retention performance using a variety of features, and discovered which were important for predicting correctness on a delayed test. We found that the strongest predictor of retention was a student's initial speed of mastering the content. The most striking finding was that students who struggled to master the content (took over 8 practice attempts) showed very poor retention, only 55% correct, after just one week. Our results will help us advance our understanding of learning and potentially improve ITS.

Schools use commercial systems specifically for mathematics benchmarking and longitudinal assessment. However these systems are expensive and their results often fail to indicate a clear path for teachers to differentiate instruction based on students’ individual strengths and weaknesses in specific skills. ASSISTments is a web-based Intelligent Tutoring System used by educators to drive real-time, formative assessment in their classrooms. The software is used primarily by mathematics teachers to deliver homework, classwork and exams to their students. We have developed a computer adaptive test called PLACEments as an extension of ASSISTments to allow teachers to perform individual student assessment and by extension school-wide benchmarking. PLACEments uses a form of graph-based knowledge representation by which the exam results identify the specific mathematics skills that each student lacks. The system additionally provides differentiated practice determined by the students’ performance on the adaptive test. In this project, we describe the design and implementation of PLACEments as a skill assessment method and evaluate it in comparison with a fixed-item benchmark.

Analyzing human learning and performance accurately is one of the main goals of an Intelligent Tutoring System. The“ASSISTment" system is a web-based system that blends assisting students and assessing their performance by providing feedback to the teachers. Good cognitive models are needed for an Intelligent Tutoring system to do a better job at predicting student performance. The ASSISTment system uses a method of cognitive modeling which is called a transfer model. A Transfer Model is a matrix that maps questions to skills. Other researchers have shown that transfer models help in building better predictive models that in-turn help in assessing a student's performance [1, 8]. They provide a viable means of representing a subject matter expert's view of which skills are needed to solve a given problem. However, the process of building a transfer model requires a lot of time. Reducing the time in which a transfer model is built would in turn help reduce the cost of building an Intelligent Tutoring System. Being able to build better transfer models will provide more efficient means of predicting learning in an intelligent tutoring system [6]. In this thesis we studied the creation of one transfer model that maps approximately the 263 released MCAS items to approximately 90 skills. Recently, [5] and [9], using two different modeling methodologies, have both concluded that this transfer model can be used to predict MCAS scores more accurately. Currently the time spent in creating and storing a model is estimated to be approximately 65 hours. This thesis was motivated by the need of a set of tools that would reduce the time spent in building a transfer model. The goal of this thesis was to create a tool that would speed up the process of building a transfer model. The efficiency of this tool is measured by an estimate of the overall time reduced for building a model. The average time reduced by using the tool on per question basis is also measured. The tool is not evaluated for its usability or for the ability to build better fitting transfer models.

Este trabalho aborda um estudo sobre os STI - Sistemas Tutores Inteligentes - dando uma visão geral do que esta sendo feito nesta área e quais são as tendências futuras que direcionam os STI a trabalhar com arquiteturas de agentes. Para simular as mudanças que ocorrem em certos estados mentais dos agentes, fizemos uma unido dos STI com a IAD - Inteligência Artificial Distribuída - e construímos os modelos dos agentes com base no ambiente dos STI e na arquitetura SEM - Sociedade dos Estados Mentais - [CORM que baseia seu formalismo na Teoria das Situações. Exploramos e adotamos a ideia da arquitetura aberta dos STI [OLI92], pois, através dela, foi possível criar um ambiente cooperativo de aprendizagem no qual o tutor e o aluno podem ensinar e aprender. Trabalhamos com dois agentes globais, a saber, o tutor e o aluno, sendo cada um deles composto por quatro agentes locais associados a determinados estados mentais do agente. Os agentes locais correspondem aos estados mentais: crença, desejo, intenção e expectativa, definidos na arquitetura SEM como agentes locais, e tratados individualmente nesta, que se preocupa com o comportamento particular de cada um. Optamos por usar a arquitetura SEM, que é uma arquitetura de agentes, no lugar de uma funcional tradicional, ou seja, composta por módulos, que é característica dos STI, porque nela podemos tratar os estados mentais como agentes locais, e assim é possível modelar o comportamento individual de cada estado e as mudanças que a interação entre os agentes provoca em cada um Abordamos três situações de ensino/aprendizagem com peculiaridades diferentes nas quais os agentes globais interagem cooperativamente com o objetivo de um ensinar o outro. Para cada dialogo, estabelecemos objetivos específicos: no primeiro, nosso interesse é na maneira como o aluno ensina uma nova estratégia ao tutor; no segundo, analisamos as mudanças das crenças do tutor sobre o conhecimento do aluno; no terceiro, nos preocupamos com as estratégias de ensino utilizadas pelo tutor. O processo de ensino/aprendizagem que acontece no desenrolar da interação entre os agentes é realizado usando o método de aprendizagem simbólica automática EBL - Explanation-Based Learning - [MIT86],[COS90] Este método proporciona a generalização do exemplo de treinamento que é incorporado as crenças e as estratégias do agente que desempenha o papel daquele que aprende, enriquecendo-as. As estratégias, que são fundamentais para os STI, são tratadas como pianos de ensino, utilizadas para promover a aprendizagem, pois definem a maneira como determinado conteúdo deve ser ensinado. Tratamos aqui as estratégias de uma maneira inovadora e diferente da tratada anteriormente [COR94]. Elas são um conjunto de ações e possuem armazenados procedimentos que são usados pelos agentes durante a interação. São determinadas e controladas conforme a intenção e usadas de acordo com as crenças, no sentido de selecionar a mais adequada para cada situação. / This study focuses on the Intelligent Tutoring System (ITS) and aims at presenting a general view concerning what has been developed in this field as well as the coming trends which lead the ITS to deal with agents' architecture. In order to simulate the changes which occur in certain mental states of the agents, we linked ITS with Distributed Artificial Intelligence (DAI) and then we built the agents' modules based on ITS environment and on SEM - Sociedade dos Estados Mentais that means Mental States Society - architecture [COR94]. Such an architecture bases its formalism on the Situation Theory. We explored and adopted the idea of the ITS open architecture [OLI92] for, through it, it has been possible to create a cooperative learning environment in which both the tutor and the student are able to teach and learn. The two global agents we worked on - tutor and student - both of them are made up of four local agents which are their mental states. The mental states involved are: belief, desire, intention, and expectation. These mental states are treated individually and defined as local agents according to SEM architecture. Instead of using a functional architecture - characteristic of ITS - we chose an agent architecture, for this latter makes it possible to treat the mental states as subagents. It is possible, therefore, to model the individual behavior of each state as well as the changes resulted from the agents' interaction. We focused on three teaching/learning situations that present different situations in which the global agents interact co-operatively in such a way that they teach each other. Specific aims were meant to each dialogue, as follows: the first dialogue concern has to do with the way the student teaches the tutor a new strategy; the second dialogue aim is to explore the tutor's "belief revision" about the student's knowledge; the third dialo gue goal has to do with the teaching strategies used by the tutor. The teaching/learning process brought about as the interaction between the agents happens is applied by using the Explanation-Based Learning (EBL) method [MIT86],[COS 90]. This method makes it possible to generalize the test example which is added to the learning agent's beliefs and strategies, making them more complete. The strategies, which are vital to the ITS, are treated as teaching plans and used to bring about learning, for they define the way in which a certain content is supposed to be taught. The strategies are treated here in a new manner, differently from the way they had formerly been [COR94]. They are a set of actions and present procedures on file that are used by the agents during the interaction. Also, the strategies are chosen and controlled by the intention and consulted by the beliefs so as to select the most suitable one, according to the situation.