University of British Columbia
Relative performance of scoring designs for the assessment of constructed responses
Abstract
dc:descriptionThis study used computer simulation to investigate the relative performance of three rater allocation designs and an objective scoring condition at two levels of comparisons: comparison to the examinee true ability, and comparison to the ability estimate obtained under the objective scoring condition. At both comparison levels, score level (theta estimate) differences across theta bins and decisions at four cut-off levels were investigated. At comparison level one, small differences across scoring designs were found in the accuracy of ability estimates obtained using the scoring designs investigated. However, large differences in percentage of accurate pass and fail decisions were found at the first comparison level. This pattern of findings was repeated at comparison level two; comparison to the objective scoring estimates and decisions. The decisions associated with the row allocation design in particular were somewhat erratic when compared to the other scoring methods at both comparison levels making it the least desirable of the rater allocation designs. The random and spiral allocation designs provided similar levels of accuracy in estimating examinee ability and provided similar levels of decision accuracy. These allocation designs showed similar levels of discrepancy and showed similar percentages of agreement with the decisions obtained under the objective scoring condition.
Degree
thesis:*- Name thesis:degree_name
- Doctor of Philosophy - PhD
- Level thesis:degree_level
- doctoral
- Discipline thesis:degree_discipline
- Measurement, Evaluation and Research Methodology
- Grantor dc:publisher
- University of British Columbia
- Year dc:date
- 2005
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Barnett, Sharon Gale
Rights
dc:rights- Statement dc:rights
-
- For non-commercial purposes only, such as research, private study and education. Additional conditions apply, see Terms of Use https://open.library.ubc.ca/terms_of_use.
- Language dc:language
- eng
Identifiers
dc:identifier.*- Handle dc:identifier
- http://hdl.handle.net/2429/17331