Back to search

University of North Texas

Reliability Generalization: a Systematic Review and Evaluation of Meta-analytic Methodology and Reporting Practice

Abstract

dc:description

Reliability generalization (RG) is a method for meta-analysis of reliability coefficients to estimate average score reliability across studies, determine variation in reliability, and identify study-level moderator variables influencing score reliability. A total of 107 peer-reviewed RG studies published from 1998 to 2013 were systematically reviewed to characterize the meta-analytic methods employed and to evaluate quality of reporting practice against standards for transparency in meta-analysis reporting. Most commonly, RG studies meta-analyzed alpha coefficients, which were synthesized using an unweighted, fixed-effects model applied to untransformed coefficients. Moderator analyses most frequently included multiple regression and bivariate correlations employing a fixed-effects model on untransformed, unweighted coefficients. Based on a unit-weighted scoring system, mean reporting quality for RG studies was statistically less than that for a comparison study of 198 meta-analyses in the organizational sciences across 42 indicators; however, means were not statistically significantly different between the two studies when evaluating reporting quality on 18 indicators deemed essential to ethical reporting practice in meta-analyses. Since its inception a wide variety of statistical methods have been applied to RG, and meta-analysis of reliability coefficients has extended to fields outside of psychological measurement, such as medicine and business. A set of guidelines for conducting and reporting RG studies is provided.

Degree

thesis:*
Grantor dc:publisher
University of North Texas
Year dc:date
2015

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Holland, David F. (Educational consultant)
Contributors dc:contributor
  • Henson, Robin K. (Robin Kyle)
  • Hull, Darrell Magness
  • Natesan, Prathiba
  • Nimon, Kim
  • Lindo, Endia J.

Subjects

dc:subject × 7

Rights

dc:rights
Statement dc:rights
  • Public
  • Holland, David F.
  • Copyright
  • Copyright is held by the author, unless otherwise noted. All rights Reserved.
Language dc:language
English

Identifiers

dc:identifier.*
Identifier
https://digital.library.unt.edu/ark:/67531/metadc822810/
ark: ark:/67531/metadc822810

Chain of custody

source
Harvested from
University of North Texas
Base URL
digital.library.unt.edu/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Holland, David F. (Educational consultant). Reliability Generalization: a Systematic Review and Evaluation of Meta-analytic Methodology and Reporting Practice. University of North Texas, 2015. https://doi.org/10.12794/metadc822810