Return to search

Reliability Generalization: a Systematic Review and Evaluation of Meta-analytic Methodology and Reporting Practice

Reliability generalization (RG) is a method for meta-analysis of reliability coefficients to estimate average score reliability across studies, determine variation in reliability, and identify study-level moderator variables influencing score reliability. A total of 107 peer-reviewed RG studies published from 1998 to 2013 were systematically reviewed to characterize the meta-analytic methods employed and to evaluate quality of reporting practice against standards for transparency in meta-analysis reporting. Most commonly, RG studies meta-analyzed alpha coefficients, which were synthesized using an unweighted, fixed-effects model applied to untransformed coefficients. Moderator analyses most frequently included multiple regression and bivariate correlations employing a fixed-effects model on untransformed, unweighted coefficients. Based on a unit-weighted scoring system, mean reporting quality for RG studies was statistically less than that for a comparison study of 198 meta-analyses in the organizational sciences across 42 indicators; however, means were not statistically significantly different between the two studies when evaluating reporting quality on 18 indicators deemed essential to ethical reporting practice in meta-analyses. Since its inception a wide variety of statistical methods have been applied to RG, and meta-analysis of reliability coefficients has extended to fields outside of psychological measurement, such as medicine and business. A set of guidelines for conducting and reporting RG studies is provided.

Identiferoai:union.ndltd.org:unt.edu/info:ark/67531/metadc822810
Date12 1900
CreatorsHolland, David F. (Educational consultant)
ContributorsHenson, Robin K. (Robin Kyle), Hull, Darrell Magness, Natesan, Prathiba, Nimon, Kim, Lindo, Endia J.
PublisherUniversity of North Texas
Source SetsUniversity of North Texas
LanguageEnglish
Detected LanguageEnglish
TypeThesis or Dissertation
Formativ, 81 pages : illustrations, Text
RightsPublic, Holland, David F., Copyright, Copyright is held by the author, unless otherwise noted. All rights Reserved.

Page generated in 0.0022 seconds