Back to search

University of British Columbia

Evaluating the correctness of IRT-based methods in computing classification consistency and accuracy indices in the presence of model misspecification

Abstract

dc:description

This thesis investigated the correctness of the classification consistency (CC) and classification accuracy (CA) indices computed using the Item Response Theory (IRT) -based Rudner, Guo, and Lee methods on tests with a single cut score for pass-or-fail decision. Specifically, simulation studies were conducted to evaluate the correctness of the classification indices under the IRT model misspecification conditions, in which a simpler, more restricted IRT model was misspecified for item calibrations. The correctness of the indices was also evaluated in the baseline conditions in which a correct IRT model was specified for item calibrations. The location of cut score on the theta scale was also manipulated to study its effects on the correctness of the indices. Overall, the results suggested that the classification indices computed from all three IRT-based methods were close to their “true” values in both model misspecification and baseline conditions when the cut score was set at or near where the test had the most information. However, it should be noted that under the model misspecification conditions, the proportions of examinees classified into either one of the binary categories (i.e. passing rates) were different from those when the correct models were specified. The finding suggests that although the computed classification indices were mostly correct when evaluated at the same theta cut score even when the IRT model was misspecified, the differences in the passing rates could have implications on the overall purpose and consequence of the classification decisions. Limitations of this thesis were also discussed.

Degree

thesis:*
Name thesis:degree_name
Master of Arts - MA
Level thesis:degree_level
master's
Discipline thesis:degree_discipline
Measurement, Evaluation and Research Methodology
Grantor dc:publisher
University of British Columbia
Year dc:date
2018

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Chau, Lok Heng

Rights

dc:rights
Statement dc:rights
  • Attribution-NonCommercial-NoDerivatives 4.0 International
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/66984

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Chau, Lok Heng. Evaluating the correctness of IRT-based methods in computing classification consistency and accuracy indices in the presence of model misspecification. master's thesis, University of British Columbia, 2018. http://hdl.handle.net/2429/66984