Back to search

University of British Columbia

A simulation study comparing the reliability and validity of methods of scoring ratings

Abstract

dc:description

Simulated rating data were generated according to a uni-factor model under varying conditions of: number of judges; number of targets; discrepancies in judges' scales of measurement; and mean and variance in distributions of individual judges' reliabilities. Burt's (1936) method of standardizing ratings, estimating judges' individual reliabilities from the rating data, and weighting ratings by a function of the judges' estimated reliabilities resulted in higher correlations with true scores than did the simple consensus. A method of scaling true score estimates to an optimal absolute scale resulted in reduced mean square deviations from the true scores. Burt's estimates showed close resemblance to maximum likelihood factor scores. Several proposed methods of estimating individual judges' reliabilities were tested. Only Cronbach's performed poorly under some conditions. The maximum likelihood factor loading estimate appeared to give the best estimate overall. The alpha coefficient was found to be a much poorer estimate of the reliability of the sum (or mean) of a group of judges than another estimate which involved estimating judges' individual reliabilities.

Degree

thesis:*
Name thesis:degree_name
Master of Arts - MA
Level thesis:degree_level
master's
Discipline thesis:degree_discipline
Psychology
Grantor dc:publisher
University of British Columbia
Year dc:date
1984

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Phillips, Norman

Rights

dc:rights
Statement dc:rights
  • For non-commercial purposes only, such as research, private study and education. Additional conditions apply, see Terms of Use https://open.library.ubc.ca/terms_of_use.
Language dc:language
eng

Identifiers

dc:identifier.*
Handle dc:identifier
http://hdl.handle.net/2429/25206

Chain of custody

source
Harvested from
University of British Columbia
Base URL
circle.library.ubc.ca/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Phillips, Norman. A simulation study comparing the reliability and validity of methods of scoring ratings. master's thesis, University of British Columbia, 1984. http://hdl.handle.net/2429/25206