Back to search

University of British Columbia

Relative performance of scoring designs for the assessment of constructed responses

Abstract

dc:description

This study used computer simulation to investigate the relative performance of three rater allocation designs and an objective scoring condition at two levels of comparisons: comparison to the examinee true ability, and comparison to the ability estimate obtained under the objective scoring condition. At both comparison levels, score level (theta estimate) differences across theta bins and decisions at four cut-off levels were investigated. At comparison level one, small differences across scoring designs were found in the accuracy of ability estimates obtained using the scoring designs investigated. However, large differences in percentage of accurate pass and fail decisions were found at the first comparison level. This pattern of findings was repeated at comparison level two; comparison to the objective scoring estimates and decisions. The decisions associated with the row allocation design in particular were somewhat erratic when compared to the other scoring methods at both comparison levels making it the least desirable of the rater allocation designs. The random and spiral allocation designs provided similar levels of accuracy in estimating examinee ability and provided similar levels of decision accuracy. These allocation designs showed similar levels of discrepancy and showed similar percentages of agreement with the decisions obtained under the objective scoring condition.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy - PhD
Level thesis:degree_level
doctoral
Discipline thesis:degree_discipline
Measurement, Evaluation and Research Methodology
Grantor dc:publisher
University of British Columbia
Year dc:date
2005

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Barnett, Sharon Gale

Rights

dc:rights
Statement dc:rights
  • For non-commercial purposes only, such as research, private study and education. Additional conditions apply, see Terms of Use https://open.library.ubc.ca/terms_of_use.
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/17331

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Barnett, Sharon Gale. Relative performance of scoring designs for the assessment of constructed responses. doctoral thesis, University of British Columbia, 2005. http://hdl.handle.net/2429/17331