University of North Texas
Reliability Generalization: a Systematic Review and Evaluation of Meta-analytic Methodology and Reporting Practice
Abstract
dc:descriptionReliability generalization (RG) is a method for meta-analysis of reliability coefficients to estimate average score reliability across studies, determine variation in reliability, and identify study-level moderator variables influencing score reliability. A total of 107 peer-reviewed RG studies published from 1998 to 2013 were systematically reviewed to characterize the meta-analytic methods employed and to evaluate quality of reporting practice against standards for transparency in meta-analysis reporting. Most commonly, RG studies meta-analyzed alpha coefficients, which were synthesized using an unweighted, fixed-effects model applied to untransformed coefficients. Moderator analyses most frequently included multiple regression and bivariate correlations employing a fixed-effects model on untransformed, unweighted coefficients. Based on a unit-weighted scoring system, mean reporting quality for RG studies was statistically less than that for a comparison study of 198 meta-analyses in the organizational sciences across 42 indicators; however, means were not statistically significantly different between the two studies when evaluating reporting quality on 18 indicators deemed essential to ethical reporting practice in meta-analyses. Since its inception a wide variety of statistical methods have been applied to RG, and meta-analysis of reliability coefficients has extended to fields outside of psychological measurement, such as medicine and business. A set of guidelines for conducting and reporting RG studies is provided.
Degree
thesis:*- Grantor dc:publisher
- University of North Texas
- Year dc:date
- 2015
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Holland, David F. (Educational consultant)
- Contributors dc:contributor
-
- Henson, Robin K. (Robin Kyle)
- Hull, Darrell Magness
- Natesan, Prathiba
- Nimon, Kim
- Lindo, Endia J.
Subjects
dc:subject × 7Rights
dc:rights- Statement dc:rights
-
- Public
- Holland, David F.
- Copyright
- Copyright is held by the author, unless otherwise noted. All rights Reserved.
- Language dc:language
- English
Identifiers
dc:identifier.*- Identifier
-
https://digital.library.unt.edu/ark:/67531/metadc822810/
ark: ark:/67531/metadc822810