An approach to education gaining popularity in the modern classroom, adaptive tutoring systems offer interactive learning environments in which students can access immediate feedback and rich tutoring while teachers can achieve organized assessment for targeted interventions. Yet despite the benefits that these systems provide, a number of questions remain regarding the optimal inner workings of adaptive platforms. What is the recipe for optimal student performance within these platforms? What elements should be taken into consideration when designing these learning environments? Can facets of these platforms be harnessed to increase students’ motivation to learn and to improve both immediate and robust learning gains? This thesis combines work conducted over the past two years through versatile approaches toward the goal of enhancing student motivation and learning within the ASSISTments platform. Approaches considered include a) enhancing motivation and performance through altered feedback using hypermedia elements, b) instilling motivational messages alongside media enhanced content and feedback, c) allowing students to choose their feedback medium, thereby exerting control over their assignment, d) altering content delivery by interleaving skills to enhance solution strategy development, and e) establishing partial credit assessments to drive motivation and proper system usage while enhancing student modeling. After a brief introduction regarding the main tenants of this research, each chapter highlights a randomized controlled trial focused around one of these approaches. All studies presented have been conducted or are still running within ASSISTments. Much of this work has already been published at peer reviewed conference venues, some with stringent acceptance rates as low as 25% for full papers. Two of the studies presented here are second iterations of previously published work that are still in progress, and only preliminary analyses are available. A chapter on conclusions and future work is included to discuss the contributions that have been made to the Learning Sciences community thus far, and to briefly discuss potential directions for my continued research.

An approach to education gaining popularity in the modern classroom, adaptive tutoring systems offer interactive learning environments in which students can access immediate feedback and rich tutoring while teachers can achieve organized assessment for targeted interventions. Yet despite the benefits that these systems provide, a number of questions remain regarding the optimal inner workings of adaptive platforms. What is the recipe for optimal student performance within these platforms? What elements should be taken into consideration when designing these learning environments? Can facets of these platforms be harnessed to increase students’ motivation to learn and to improve both immediate and robust learning gains? This thesis combines work conducted over the past two years through versatile approaches toward the goal of enhancing student motivation and learning within the ASSISTments platform. Approaches considered include a) enhancing motivation and performance through altered feedback using hypermedia elements, b) instilling motivational messages alongside media enhanced content and feedback, c) allowing students to choose their feedback medium, thereby exerting control over their assignment, d) altering content delivery by interleaving skills to enhance solution strategy development, and e) establishing partial credit assessments to drive motivation and proper system usage while enhancing student modeling. After a brief introduction regarding the main tenants of this research, each chapter highlights a randomized controlled trial focused around one of these approaches. All studies presented have been conducted or are still running within ASSISTments. Much of this work has already been published at peer reviewed conference venues, some with stringent acceptance rates as low as 25% for full papers. Two of the studies presented here are second iterations of previously published work that are still in progress, and only preliminary analyses are available. A chapter on conclusions and future work is included to discuss the contributions that have been made to the Learning Sciences community thus far, and to briefly discuss potential directions for my continued research.

Die vorliegende Studie diskutiert die Entwicklung eines Testinstruments zur Messung von Bewertungskompetenz im Sinne der Teilkompetenz Bewerten, Entscheiden und Reflektieren (BER) innerhalb des Göttinger Modells der Bewertungskompetenz im Kontext nachhaltiger Entwicklung (Bögeholz 2011) für das Unterrichtsfach Physik in der Sekundarstufe. Die ausgewählten Aufgabenkontexte beschreiben die Erzeugung, die Speicherung und die Nutzung elektrischer Energie. Sie schließen damit auch an die aktuelle gesellschaftliche Diskussion um Erneuerbare Energien an und untersuchen diesbezüglich das Entscheidungsvermögen und die Bewertungskompetenz heutiger Schülerinnen und Schüler. Die Einsatzfähigkeit des in dieser Studie entwickelten Testinstruments wurde zunächst im Rahmen zweier Vorstudien überprüft, bevor die Haupterhebung als Querschnittstudie in den Jahrgängen 6, 8, 10 und 12 erfolgte (N = 850 Schülerinnen und Schüler an Gymnasien). Nach dem Ansatz von Eggert (2008), Eggert und Bögeholz (2006, 2010) ist es dabei als paper-and-pencil -Test konzipiert und beinhaltet zwei Entscheidungsaufgaben und eine Reflexionsaufgabe. Die empirisch gewonnenen Daten wurden zunächst anhand eines entwickelten Scoring Guides codiert und anschließend sowohl unter Gesichtspunkten der Klassischen als auch der Probabilistischen Testtheorie ausgewertet. Das entwickelte Testinstrument hat sich unter Reliabilitäts- und Validitätsaspekten bewährt. Item-Fit-Parameter zeigen, dass sich die empirischen Daten gut in einem eindimensionalen Rasch-Partial-Credit-Modell abbilden lassen. Unter anderem konnten Zusammenhänge von BER mit dem Schulalter der Schülerinnen und Schüler nachgewiesen werden. Geringe Korrelationen von BER bestehen zu verschiedenen Schulnoten (u. a. zu Deutsch, Mathematik, Politik und Physik in der Klasse 10), zudem wird das Testergebnis für BER kaum von Lesekompetenzen beeinflusst. <p><p> Externer Link zum Testinstrument: http://dx.doi.org/10.7477/39:41:17

To enhance test security of high stakes tests, it is vital to understand the way various exposure control strategies function under various IRT models. To that end the present dissertation focused on the performance of several exposure control strategies under the generalized partial credit model with an item pool of 100 and 200 items. These procedures are relatively easy to implement and have shown promise as an alternative to more complex exposure control strategies. Through unique algorithms these procedures select an item for administration from a subset of items in the item pool. The five procedures examined for efficacy were the modified within .10 logits, restricted modified within .10 logits, randomesque, restricted randomesque, and progressive restricted procedures. The modified within .10 logits, restricted modified within .10 logits, and randomesque, and restricted randomesque procedures select an item for administration from a subset of optimal items. To test the effect of the number of items available for selection in this subset, 3, 6, and 9 items were made available for selection in these procedures. Maximum information item selection was used as a base line, no exposure control, condition. The progressive restricted, restricted randomesque, and restricted modified within .10 logits procedures were found to optimally protect test security while not significantly degrading measurement precision. The restricted forms of the randomesque and modified within .10 logits procedures proved superior to their base procedures, particularly in controlling average maximum exposure rate. The incrementation of item group size in the modified within .10 logits, restricted modified within .10 logits, and randomesque, and restricted randomesque procedures demonstrated that increasing the item group size provided better test security while not significantly degrading measurement precision. Additionally, in general, the increase of the item pool size from 100 to 200 improved measurement precision and test security. Implications towards practical application are discussed and directions for future research are suggested. / text

Information Systems (IS) research frequently uses survey data to measure the interplay between technological systems and human beings. Researchers have developed sophisticated procedures to build and validate multi-item scales that measure real world phenomena (latent constructs). Most studies use the so-called classical test theory (CTT), which suffers from several shortcomings. We first compare CTT to Item Response Theory (IRT) and subsequently apply a Rasch model approach to measure hedonic aspects of websites. The results not only show which attributes are best suited for scaling hedonic information systems, but also introduce IRT as a viable substitute that overcomes severall shortcomings of CTT. (author´s abstract) / Series: Research Report Series / Department of Statistics and Mathematics

Although the design of Multistage testing (MST) has received increasing attention, previous studies mostly focused on comparison of the psychometric properties of MST with CAT and paper-and-pencil (P&P) test. Few studies have systematically examined the number of items in the routing test, the number of subtests in a stage, or the number of stages in a test design to achieve accurate measurement in MST. Given that none of the studies have identified an ideal MST test design using polytomously-scored items, the current study conducted a simulation to investigate the optimal design for MST using generalized partial credit model (GPCM). Eight different test designs were examined on ability estimation across two routing test lengths (short and long) and two total test lengths (short and long). The item pool and generated item responses were based on items calibrated from a national test consisting of 273 partial credit items. Across all test designs, the maximum information routing method was employed and the maximum likelihood estimation was used for ability estimation. Ten samples of 1,000 simulees were used to assess each test design. The performance of each test design was evaluated in terms of the precision of ability estimates, item exposure rate, item pool utilization, and item overlap. The study found that all test designs produced very similar results. Although there were some variations among the eight test structures in the ability estimates, results indicate that the performance overall of these eight test structures in achieving measurement precision did not substantially deviate from one another with regard to total test length and routing test length. However, results from the present study suggest that routing test length does have a significant effect on the number of non-convergent cases in MST tests. Short routing tests tended to result in more non-convergent cases, and the presence of fewer stage tests yielded more of such cases than structures with more stages. Overall, unlike previous findings, the results of the present study indicate that the MST test structure is less likely to be a factor impacting ability estimation when polytomously-scored items are used, based on GPCM. / text

Computerized adaptive testing (CAT) provides a highly efficient alternative to the paper-and-pencil test. By selecting items that match examinees’ ability levels, CAT not only can shorten test length and administration time but it can also increase measurement precision and reduce measurement error. In CAT, maximum information (MI) is the most widely used item selection procedure. However, the major challenge with MI is the attenuation paradox, which results because the MI algorithm may lead to the selection of items that are not well targeted at an examinee’s true ability level, resulting in more errors in subsequent ability estimates. The solution is to find an alternative item selection procedure or an appropriate ability estimation method. CAT studies have not investigated the association between these two components of a CAT system based on polytomous IRT models. The present study compared the performance of four item selection procedures (MI, MPWI, MEI, and MEPV) across four ability estimation methods (MLE, WLE, EAP-N, and EAP-PS) under the mixed-format CAT based on the generalized partial credit model (GPCM). The test-unit pool and generated responses were based on test-units calibrated from an operational national test that included both independent dichotomous items and testlets. Several test conditions were manipulated: the unconstrained CAT as well as the constrained CAT in which the CCAT was used as the content-balancing, and the progressive-restricted procedure with maximum exposure rate equal to 0.19 (PR19) served as the exposure control in this study. The performance of various CAT conditions was evaluated in terms of measurement precision, exposure control properties, and the extent of selected-test-unit overlap. Results suggested that all item selection procedures, regardless of ability estimation methods, performed equally well in all evaluation indices across two CAT conditions. The MEPV procedure, however, was favorable in terms of a slightly lower maximum exposure rate, better pool utilization, and reduced test and selected-test-unit overlap than with the other three item selection procedures when both CCAT and PR19 procedures were implemented. It is not necessary to implement the sophisticated and computing-intensive Bayesian item selection procedures across ability estimation methods under the GPCM-based CAT. In terms of the ability estimation methods, MLE, WLE, and two EAP methods, regardless of item selection procedures, did not produce practical differences in all evaluation indices across two CAT conditions. The WLE method, however, generated significantly fewer non-convergent cases than did the MLE method. It was concluded that the WLE method, instead of MLE, should be considered, because the non-convergent case is less of an issue. The EAP estimation method, on the other hand, should be used with caution unless an appropriate prior θ distribution is specified. / text

The aim of the monograph is to contribute towards bridging the gap between methodological developments that have evolved in the social sciences, in particular in psychometric research, and methods of statistical modelling in a more general framework. The first part surveys certain special psychometric models (often referred to as Rasch family of models) that share common properties: separation of parameters describing qualities of the subject under investigation and parameters related to properties of the situation under which the response of a subject is observed. Using conditional maximum likelihood estimation, both types of parameters may be estimated independently from each other. In particular, the Rasch model, the rating scale model, the partial credit model, hybrid types, and linear extensions thereof are treated. The second part reviews basic ideas of generalized linear models (GLMs) as an an excellent framework for unifying different approaches and providing a natural, technical background for model formulation, estimation and testing. This is followed by a short introduction to the software package GLIM chosen to illustrate the formulation of psychometric models in the GLM framework. The third part is the main part of this monograph and shows the application of generalized linear models to psychometric approaches. It gives a unified treatment of Rasch family models in the context of log-linear models and contains some new material on log-linear longitudinal modelling. The last part of the monograph is devoted to show the usefulness of the latent variable approach in a variety of applications, such as panel, cross-over, and therapy evaluation studies, where standard statistical analysis does not necessarily lead to satisfactory results. (author´s abstract) / Series: Research Report Series / Department of Statistics and Mathematics

This study reports the development of an instrument intended to measure mathematics teachers' knowledge of Cognitively Guided Instruction (CGI). CGI is a mathematics professional development framework based on how students think about and solve problems and how that knowledge guides instruction for developing mathematical understanding. The purpose of this study was to (a) analyze and revise the original CGI Teacher Knowledge Assessment (CGI TKA), (b) administer the revised CGI TKA, and (c) analyze the results from the revised CGI TKA. As part of the revision of the original CGI TKA, distractor analysis identified distractors that could be improved. Experts in CGI content were interviewed to identify ways in which the content of the CGI TKA could be improved, and some new items were created based on their feedback. Formatting changes were also made to administer the assessment electronically.After the original CGI TKA was revised, the revised CGI TKA was administered to teachers who had been trained in CGI. Two hundred thirteen examinees completed the revised CGI TKA and the results were analyzed. Exploratory and confirmatory factor analyses showed 21 of the items loaded adequately onto one factor, considered to be overall knowledge of CGI. The Rasch model was used to estimate item difficulty and person abilities as well as to compare models using dichotomous and partial credit scoring. Advantages and disadvantages of using partial credit scoring as compared to dichotomous scoring are discussed. Except under special circumstances, the dichotomous scoring produced better fitting models and more reliable scores than the partial credit scoring. The reliability of the scores was estimated using Raykov's rho coefficient. Overall, the revised CGI TKA appears to validly and reliably measure teachers' CGI knowledge.

Wie die meisten westlichen Streitkräfte, bewegt sich die Bundeswehr im Spannungsfeld zwischen hohem Personalbedarf und Fachkräftemangel. Durch ein Onlineassessment kann der Bewerbungsprozess dahingehend optimiert werden, dass fähiges Personal schneller gebunden wird. Onlineassessment hat diverse Vorteile, gleichzeitig sind damit jedoch Herausforderungen verbunden. Die wahrscheinlich größte ist es, Betrug zu minimieren, da Onlineassessment in einer weitestgehend unkontrollierten Umgebung stattfindet. Zur Entgegnung dieser Problematik dienen verschiedene Ansätze, wie beispielsweise große Itempools, wodurch einer Verbreitung der Lösung im Internet entgegengewirkt werden kann. Dieser Ansatz ist jedoch mit hohen Kosten verbunden. Automatische Itemgenerierung hingegen ermöglicht es, kostengünstig und zeiteffizient psychometrisch hochwertige Items zu erstellen. Aus diesem Grund wurden in der vorliegenden Arbeit zwei Arbeitsgedächtnistests mit automatischer Itemgenerierung für das Onlineassessment der Bundeswehr entwickelt und evaluiert, mit dem Ziel einer hohen prädiktiven Validität auf die Diagnostik vor Ort. In der ersten Studie (N = 330) wurde gezeigt, dass automatische Itemgenerierung für die entwickelten Arbeitsgedächtnistests eingesetzt werden kann. Hierbei wurden zudem zwei verschiedene zeitliche Varianten untersucht, wobei sich diejenige mit der längeren Stimulusrepräsentationszeit als vorteilhafter erwies. In der zweiten Studie (N = 621) wurden Nachweise für Reliabilität und Validität erbracht. Die Tests zeigten eine gute konvergente und diskriminante Validität. Zudem konnte einer der beiden Tests eine sehr gute prädiktive Validität aufweisen. Unter Gesamtberücksichtigung der Testgütekriterien wurde dieser Test schließlich für das Onlineassessment der Bundeswehr vorgeschlagen. Somit steht der Bundeswehr nun ein wissenschaftlich fundierter Arbeitsgedächtnistest für das Onlineassessment zur Verfügung. / Like most Western armed forces, the Bundeswehr faces both high personnel requirements and a shortage of skilled personnel. Online assessment can optimize the application process to ensure that capable personnel are retained more quickly. Online assessment has various advantages, but also challenges associated with it. Probably the biggest of these challenges is minimizing cheating, as online assessment takes place in a largely unsupervised environment. Various approaches are used to counter this problem, such as large item pools, which can be used to counter the dissemination of solutions on the Internet. However, this approach is associated with high costs. Automatic item generation, on the other hand, makes it possible to create psychometrically high-quality items in a cost-effective and time-efficient manner. For this reason, two working memory tests with automatic item generation for the German Armed Forces’ online assessment were developed and evaluated in the present study, with the aim of matching the high predictive validity of on-site diagnostics. The first study (N = 330) demonstrated that automatic item generation can be used for the developed working memory tests. Two different temporal variants were also investigated, with the longer stimulus presentation time proving to be more beneficial. The second study (N = 621) provided reliability and validity evidence. The tests showed good convergent and discriminant validity. In addition, one of the two tests demonstrated very good predictive validity. Taking into account the overall test quality criteria, this test was ultimately proposed for use in the German Armed Forces’ online assessment. Thus, the Bundeswehr now has a scientifically-grounded working memory test available for its online assessment.