The need for formative assessments has led to the development of a psychometric framework known as diagnostic classification models (DCMs), which are mathematical measurement models designed to estimate the possession or mastery of a designated set of skills or attributes within a chosen construct. Furthermore, much research has gone into the practice of “retrofitting” diagnostic measurement models to existing assessments in order to improve their diagnostic capability. Although retrofitting DCMs to existing assessments can theoretically improve diagnostic potential, it is also prone to challenges including identifying multidimensional traits from largely unidimensional assessments, a lack of assessments that are suitable for the DCM framework, and statistical quality, specifically highly correlated attributes and poor model fit. Another recent trend in assessment has been a move towards creating more authentic constructed-response assessments. For such assessments, rubric-based scoring is often seen as method of providing reliable scoring and interpretive formative feedback. However, rubric-scored tests are limited in their diagnostic potential in that they are usually used to assign unidimensional numeric scores. It is the purpose of this thesis to propose general methods for retrofitting DCMs to rubric-scored assessments. Two methods will be proposed and compared: (1) automatic construction of an attribute hierarchy to represent all possible numeric score levels from a rubric-scored assessment and (2) using rubric criterion score level descriptions to imply an attribute hierarchy. This dissertation will describe these methods, discuss the technical and mathematical issues that arise in using them, and apply and compare both methods to a prominent rubric-scored test of critical thinking skills, the Collegiate Learning Assessment+ (CLA+). Finally, the utility of the proposed methods will be compared to a reasonable alternative methodology: the use of polytomous IRT models, including the Graded Response Model (GRM), the Partial Credit Model (PCM), and the Generalized-Partial Credit Model (G-PCM), for this type of test score data.

李晓煦. / Submitted: Jan. 2010. / Thesis (doctoral)--Chinese University of Hong Kong, 2010. / Includes bibliographical references (p. 170-175). / Electronic reproduction. Hong Kong : Chinese University of Hong Kong, [2012] System requirements: Adobe Acrobat Reader. Available via World Wide Web. / Abstracts in Chinese and English. / Li Xiaoxu.

Latent variable models play an important role in educational and psychological measurement, where items are presented to individuals, resulting in item response data. Such data entail important information about the individual latent traits, population structure and item design, which are key components to be understood in educational and psychological assessments. This thesis focuses on the development of statistical learning methods based on latent variable models with identifiability theories. The thesis consists of three parts, with three kinds of applications in mind. The first part is on the identifiability of diagnostic classification models (DCMs), which is a special subfamily of latent class models. It aims to examine the test takers' ability based on his/her mastery of set of required skills. A key issue common to DCMs and more generally to latent class model is the identifiability which is a property whether the unknown model or related parameters can be estimated consistently under a suitably defined asymptotic regime. Most existing works focus on the identifiability of DCMs with binary responses and attributes. In this thesis, we provide general identifiability results for DCMs with polytoumous responses and attributes and less parameter restrictions. The second part considers the identifiability of testlet factor models, which is a subfamily of latent variable models with underlying continuous latent traits assumed to follow normal distribution. Similar to DCMs, factor models also suffer from identifiability issues, where the parameters can only be identified up to a rotation in general. However, in most applications, testlet models or bifactor models are popular in educational assessment. They are constrained factor models assuming that the response test items can be accounted for by one primary factor and multiple secondary group-specific factors. By aid of this special structure, we can show that the model can be strictly identifiable and we provide checkable necessary and sufficient conditions accordingly. The third part focuses on the statistical learning in studying the complex problem-solving (CPS) items. With advanced computer technology, there is a new trend of developing CPS test items through online platform, where the examinees are asked to solve challenging tasks in a simulated environment. During the test, all actions performed by examinees will be recored into a log file. Therefore, we can not only observe their final responses, but also have access to their entire solving process. Such data type is known as process data in the measurement literature. The traditional item response model cannot be applicable, at least directly. The analysis of the process data is still in its infancy. In the thesis, we propose a new model-based approach and show its usefulness through an interesting real data application.

With advances in computer technology and expanded access to educational data, psychometrics faces new opportunities and challenges for enhancing pattern discovery and decision-making in testing and learning. In this dissertation, I introduced three computational psychometrics studies for solving the technical problems in item-based computerized adaptive learning (CAL) systems related to dynamic measurement, diagnosis, and recommendation based on Bayesian item response theory (IRT). For the first study, I introduced a new knowledge tracing (KT) model, dynamic IRT (DIRT), which can iteratively update the posterior distribution of latent ability based on moment match approximation and capture the uncertainty of ability change during the learning process. For dynamic measurement, DIRT has advantages in interpretation, flexibility, computation cost, and implementability. For the second study, A new measurement model, named multilevel and multidimensional item response theory with Q matrix (MMIRT-Q), was proposed to provide fine-grained diagnostic feedback. I introduced sequential Monte Carlo (SMC) for online estimation of latent abilities. For the third study, I proposed the maximum expected ratio of posterior variance reduction criterion (MERPV) for testing purposes and the maximum expected improvement in posterior mean (MEIPM) criterion for learning purposes under the unified framework of IRT. With these computational psychometrics solutions, we can improve the students’ learning and testing experience with accurate psychometrics measurement, timely diagnosis feedback, and efficient item selection.