Collaboration among teachers in some shape or form is becoming increasing popular among the educational system. The goal of this thesis is to determine whether teachers find value in collaboratively working in a Web environment and if we can use collaboration to improve educational content. We took a Web-based intelligent tutoring system, called ASSISTments, and incorporated a collaboration feature allowing teachers from around the Web to work together to create content for their students. The previous ASSISTments model did not allow for any form of collaboration; teachers using ASSISTments were not able to modify each other's content. By creating the opportunity for teachers to work together, we hypothesized that the educational content within ASSISTments would improve. To help improve education content among ASSISTments, we also deemed it necessary to improve the tool that teachers used to create problems for their students. Using surveys and interviews, we obtained feedback from teachers supporting our changes of the ASSISTments system and validating our claims that they found value in collaboratively working in a Web-based environment.

It is the underlying presupposition of the Science Assistments research (http://www.scienceassistments.org) that students need to leave school with a basic understanding of science and grounding in inquiry skills (NSES, 1996; NRC, 2011). We also believe that the current standard for assessing these skills, the Massachusetts Comprehensive Assessment System, is inadequate in terms of the rote- oriented multiple-choice tests. This thesis describes the creation of a simulation, or microworld, of an animal cell. This content is aligned with the Massachusetts science frameworks for middle school Life Science (Massachusetts Department of Education, 2006). Our microworld, Simcell, gives students an opportunity to form hypotheses, design experiments to test these hypotheses, and analyze their data collected during the experiment. The microworlds track students' actions in log files that can be analyzed by the system to provide fine tuned assessments of students, and based on these assessments, in the future, we will provide dynamic help though scaffolds to students who are struggling with inquiry (Gobert et al, 2007; 2009; Gobert et al, in press). Over the course of two studies, this biology microworld was designed, developed, and fined tuned through the use of domain experts and student pilot data. We also analyzed the student logs in order to try to model students' learning so we can predict useful times for the system to come in and help. In study one we identify a potential point to remediate struggling students. In study two we conducted a series of logistic and linear regressions to predict student knowledge. However, due to the large number of different variables and the relatively small size of the dataset, we could not be confident in the results that were obtained. Many attempts to reduce the number of variables used in the model were tried, but these methods did not yield more promise than the original set. Finally, we finish this report with a new path for researchers to consider, namely, looking at the data in different ways in order to find a way of viewing the data that would allow for known successful student modeling techniques such as Bayesian Knowledge Tracing.

ASSISTments is an online adaptive tutoring system with the ability to provide assistance to students in the form of hints and scaffolding. ASSISTments has many features to help students improve their knowledge. Researchers run studies in order to discover ways for students to learn better but ASSISTments is missing one major aspect for researchers: student level personalization. It is easy to create an assignment for a particular class or school but it would take much longer to create an assignment for each student and it would be difficult for the teacher to look through many assignment reports. One of the strongest code blocks in coding is the if-then; allowing the program to branch off to another set of code under certain circumstances. ASSISTments needed an if-then system in order for students to branch off to other parts of the assignment under certain circumstances. With this, researchers would be able to personalize assignments to give more help to lower knowledge students or allow students to get a choice of what kind of tutoring they would like to receive. With this idea in mind, the basic if-then structure was implemented into ASSISTments using problem or problem set correctness as the condition statement. Once the if-then system was created opportunities opened to create additional experiments and run studies in ASSISTments. The basic if-then was limited in using correctness only for its condition statement. This meant that a new if-then system would need to be implemented to include custom condition statements that allowed the researcher to have the assignment branch on any condition using all the information recorded in the assignment. While work was being done on the if-then system, research was being done and two papers were written on partial credit in ASSISTments. Partial credit was found out to be as accurate as knowledge tracing in determining student performance on the next problem. Once a partial credit algorithm was found, a study using if-then was analyzed. It was found that there was no statistically significant difference between students who were given a choice on their feedback and students who received no choice.

This study explores the ability of an Intelligent Tutoring System (ITS) to increase parental engagement in student learning. A parental notification feature was developed for the web-based ASSISTments ITS that allows parents to log into their own accounts and access detailed data about their students' performance. Parents from a local middle school were then invited to create accounts and answer a survey assessing how engaged they felt they were in their students' education. A randomized controlled experiment was run during which weekly automated messages were sent home to parents regarding their students' assignments and how they were performing. After having them take a post-survey, it was found that access to this data caused parents to become more involved in their students' education. Additionally, this led to increased student performance in the form of higher homework completion rates. Qualitative feedback from parents was very positive.

An approach to education gaining popularity in the modern classroom, adaptive tutoring systems offer interactive learning environments in which students can access immediate feedback and rich tutoring while teachers can achieve organized assessment for targeted interventions. Yet despite the benefits that these systems provide, a number of questions remain regarding the optimal inner workings of adaptive platforms. What is the recipe for optimal student performance within these platforms? What elements should be taken into consideration when designing these learning environments? Can facets of these platforms be harnessed to increase students’ motivation to learn and to improve both immediate and robust learning gains? This thesis combines work conducted over the past two years through versatile approaches toward the goal of enhancing student motivation and learning within the ASSISTments platform. Approaches considered include a) enhancing motivation and performance through altered feedback using hypermedia elements, b) instilling motivational messages alongside media enhanced content and feedback, c) allowing students to choose their feedback medium, thereby exerting control over their assignment, d) altering content delivery by interleaving skills to enhance solution strategy development, and e) establishing partial credit assessments to drive motivation and proper system usage while enhancing student modeling. After a brief introduction regarding the main tenants of this research, each chapter highlights a randomized controlled trial focused around one of these approaches. All studies presented have been conducted or are still running within ASSISTments. Much of this work has already been published at peer reviewed conference venues, some with stringent acceptance rates as low as 25% for full papers. Two of the studies presented here are second iterations of previously published work that are still in progress, and only preliminary analyses are available. A chapter on conclusions and future work is included to discuss the contributions that have been made to the Learning Sciences community thus far, and to briefly discuss potential directions for my continued research.

An approach to education gaining popularity in the modern classroom, adaptive tutoring systems offer interactive learning environments in which students can access immediate feedback and rich tutoring while teachers can achieve organized assessment for targeted interventions. Yet despite the benefits that these systems provide, a number of questions remain regarding the optimal inner workings of adaptive platforms. What is the recipe for optimal student performance within these platforms? What elements should be taken into consideration when designing these learning environments? Can facets of these platforms be harnessed to increase students’ motivation to learn and to improve both immediate and robust learning gains? This thesis combines work conducted over the past two years through versatile approaches toward the goal of enhancing student motivation and learning within the ASSISTments platform. Approaches considered include a) enhancing motivation and performance through altered feedback using hypermedia elements, b) instilling motivational messages alongside media enhanced content and feedback, c) allowing students to choose their feedback medium, thereby exerting control over their assignment, d) altering content delivery by interleaving skills to enhance solution strategy development, and e) establishing partial credit assessments to drive motivation and proper system usage while enhancing student modeling. After a brief introduction regarding the main tenants of this research, each chapter highlights a randomized controlled trial focused around one of these approaches. All studies presented have been conducted or are still running within ASSISTments. Much of this work has already been published at peer reviewed conference venues, some with stringent acceptance rates as low as 25% for full papers. Two of the studies presented here are second iterations of previously published work that are still in progress, and only preliminary analyses are available. A chapter on conclusions and future work is included to discuss the contributions that have been made to the Learning Sciences community thus far, and to briefly discuss potential directions for my continued research.

The goal of this thesis is to demonstrate how user testing can be used to identify and remediate learnability issues of a web application. Experimentation revolved around ASSISTments (www.assistments.org), an intelligent tutoring web application in which teachers create virtual classrooms where they can assign problem sets to their students and gain valuable data which can be used to make informed decisions. Recent log analysis uncovered very low task completion rates for new users on tasks that were intended to be trivial. Suspecting that this could be due to poor user interface design, user tests were conducted to help identify usability problems. Sessions were analyzed, and changes were made between each user test to address issues found. Feedback from user testing led to the implementation of an embedded support system. This support system consisted of a splash page which gave an overview of how the system should be used and a collection of context-sensitive tooltips which tried to give the user instructions on what to do as well as explain various parts of the interface. A randomized control trial was performed to measure the effectiveness of the embedded support. 69 participants were shown one of two interfaces: one with embedded support and one without. Task completion rates were analyzed for each of the groups. We found that the support system was able to influence which links a user clicked. However, although the support system was intended to address poor task completion rates, users in the conditions had similar task completion rates regardless of whether the support system was enabled.

"Object-Oriented Programming (OOP), in the past two decades, has become the most influential and dominant programming paradigm for developing large and complex software systems. With OOP, developers can rely on design patterns that are widely accepted as solutions for recurring problems and used to develop flexible, reusable and modular software. However, recent studies have shown that Objected-Oriented Abstractions are not able to modularize these pattern concerns and tend to lead to programs with poor modularity. Feature-Oriented Programming (FOP) is an extension of OOP that aims to improve the modularity and to support software variability in OOP by refining classes and methods. In this thesis, based upon the work of integrating an online tutor systems, ASSISTments, with other online learning management systems, we evaluate FOP with respect to modularity. This proof-of-concept effort demonstrates how to reduce the effort in designing integration code."

With the wide usage of online tutoring systems, researchers become interested in mining data from logged files of these systems, so as to get better understanding of students. Varieties of aspects of students’ learning have become focus of studies, such as modeling students’ mastery status and affects. On the other hand, Randomized Controlled Trial (RCT), which is an unbiased method for getting insights of education, finds its way in Intelligent Tutoring System. Firstly, people are curious about what kind of settings would work better. Secondly, such a tutoring system, with lots of students and teachers using it, provides an opportunity for building a RCT infrastructure underlying the system. With the increasing interest in Data mining and RCTs, the thesis focuses on these two aspects. In the first part, we focus on analyzing and mining data from ASSISTments, an online tutoring system run by a team in Worcester Polytechnic Institute. Through the data, we try to answer several questions from different aspects of students learning. The first question we try to answer is what matters more to student modeling, skill information or student information. The second question is whether it is necessary to model students’ learning at different opportunity count. The third question is about the benefits of using partial credit, rather than binary credit as measurement of students’ learning in RCTs. The fourth question focuses on the amount that students spent Wheel Spinning in the tutoring system. The fifth questions studies the tradeoff between the mastery threshold and the time spent in the tutoring system. By answering the five questions, we both propose machine learning methodology that can be applied in educational data mining, and present findings from analyzing and mining the data. In the second part, we focused on RCTs within ASSISTments. Firstly, we looked at a pilot study of reassessment and relearning, which suggested a better system setting to improve students’ robust learning. Secondly, we proposed the idea to build an infrastructure of learning within ASSISTments, which provides the opportunities to improve the whole educational environment.

The complexities of how people learn have plagued researchers for centuries. A range of experimental and non-experimental methodologies have been used to isolate and implement positive interventions for students' cognitive, meta-cognitive, behavioral, and socio-emotional successes in learning. But the face of learning is changing in the digital age. The value of accrued knowledge, popular throughout the industrial age, is being overpowered by the value of curiosity and the ability to ask critical questions. Most students can access the largest free collection of human knowledge (and cat videos) with ease using their phones or laptops and omnipresent cellular and Wi-Fi networks. Viewing this new-age capacity for connection as an opportunity, educational stakeholders have delegated many traditional learning tasks to online environments. With this influx of online learning, student errors can be corrected with immediacy, student data is more prevalent and actionable, and teachers can intervene with efficiency and efficacy. As such, endeavors in educational data mining, learning analytics, and authentic educational research at scale have grown popular in recent years; fields afforded by the luxuries of technology and driven by the age-old goal of understanding how people learn. This dissertation explores the evolution and application of ASSISTments Research, an approach to authentic educational research at scale that leverages ASSISTments, a popular online learning platform, to better understand how people learn. Part I details the evolution and advocacy of two tools that form the research arm of ASSISTments: the ASSISTments TestBed and the Assessment of Learning Infrastructure (ALI). An NSF funded Data Infrastructure Building Blocks grant (#1724889, $494,644 2017-2020), outlines goals for the new age of ASSISTments Research as a result of lessons learned in recent years. Part II details a personal application of these research tools with a focus on the framework of Self Determination Theory. The primary facets of this theory, thought to positively affect learning and intrinsic motivation, are investigated in depth through randomized controlled trials targeting Autonomy, Belonging, and Competence. Finally, a synthesis chapter highlights important connections between Parts I & II, offering lessons learned regarding ASSISTments Research and suggesting additional guidance for its future development, while broadly defining contributions to the Learning Sciences community.