Divergent thinking (DT) tasks that solicit responses to open-ended stimuli are the most common form of assessment used in creativity research. Intended to measure creative potential, these tasks present researchers with multiple choices throughout administration and scoring that often result in widely varying scores. This study used a combined dataset from 9 independent samples (N = 1,066) containing DT responses (n = 15,935) to the alternate uses task to evaluate substantive validity under different scoring approaches. Human ratings of creative quality were compared to 11 additional quality scores based on statistical infrequency and semantic distance. Psychometric analyses were supplemented by review of response content to assess conceptual and operational correspondence. Results revealed several inconsistencies between and within scoring approaches, including considerable subjectivity required to group semantically equivalent responses before applying objective frequency rules; low reliability estimates for statistically unique responses; dictionary uses for some prompts scored as unique under 5% and 10% infrequency thresholds; variation in semantic distance scores of responses with similar meanings; and large within-subject differences across scoring types. Substantive review also indicated that how the creative dimension of appropriateness is operationalized can play a sizable role in score differences, such as treatment of responses featuring randomness, humor, fantasy, or cultural references. Recommendations for expanded reporting practices to support substantive validity in future DT research are discussed.