Back to search

Michigan State University

Examining alignment indices' validity as measures of test content representativeness

Abstract

dc:description

Alignment index values are often presented as evidence that test content is representative of the performance domain defined by a written curriculum. Alignment measures bear on the validity of state achievement test score interpretations, and on test fairness. While alignment reports have been used to document test content distribution, and to generate recommendations for test form improvement that appear sensible to assessment professionals, there is little external evidence that alignment index values are valid quantitative measures of tests’ content representativeness. Using eleven states' mathematics achievement test item and Surveys of Enacted Curriculum (SEC; Porter, 2002) content analysis data, I examine external validation evidence for the curriculum emphasis measures underlying the coarse-grained SEC test-curriculum alignment index. I then use fractional logit regression models to assess the relationship between state-level test-curriculum alignment and proportion-correct item difficulty on corresponding test items, controlling for other state and test item characteristics that may affect achievement. I find that the content analysis proportions that summarize SEC alignment panelists' judgments about curriculum objectives' topics and cognitive demand requirements seem to relate in expected ways to other measures of state curricular emphasis in Grade 4, providing weak external validation support for the mathematics curriculum content analysis data. However, there is no evidence of a statistically or substantively significant relationship between an alignment measure based on SEC coarse-grained content analysis data, and test item difficulty in Grade 8, regardless of the extent to which a particular item's content type is emphasized by the curriculum. I conclude that although external validation studies for the SEC alignment index have tended to focus on the relationship between curricular alignment and test performance, other types of evidence may yield clearer conclusions, and perhaps be more crucial for demonstrating these indices' validity. Specifically, I suggest future research to evaluate the content classification schemes implemented by popular alignment methods, and to establish the reproducibility of overall and item-level alignment results across independent panels of qualified experts trained by different facilitators.

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Traynor, Anne
Contributors dc:contributor
  • Reckase, Mark D.
  • Floden, Robert E.
  • Houang, Richard T.
  • McCrory, Raven

Subjects

dc:subject × 3

Rights

dc:rights
Statement dc:rights
  • In Copyright
Language dc:language
English

Identifiers

dc:identifier.*
Identifier
etd:2835
isbn:9781303960451
isbn:1303960451
oclc:973328656
umi:3623528
local:Traynor_grad.msu_0128D_12961

Chain of custody

source
Harvested from
Michigan State University
Base URL
d.lib.msu.edu/oai
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Traynor, Anne. Examining alignment indices' validity as measures of test content representativeness. 2014. https://doi.org/doi:10.25335/7kh3-5s63