Problem. This study assessed the predictive validity of criterion-referenced tests in a military setting with cutoff scores set by the Angoff and conventional score-setting methods.;Procedure. Thirty-six instructors and thirty-six specialists assessed each test item for job relevance and the probability that a minimally competent person would answer each question correctly, resulting in a new test cutoff score. Intragroup variability and interrater reliability of judgments were calculated. Test predictive validity assessment compared classroom test scores, supervisory rating scores, and skill qualification test scores of 100 job performers based on the two score-setting methods. Sample sizes varied from 17 to 100. Behaviorally anchored rating scale was used to estimate soldier performance effectiveness. Hypotheses were tested using analysis of variance, a correlation procedure by Ebel, t-test, and Pearson Product-Moment correlation. Null was accepted or rejected at.05 level of significance.;Results. Findings follow: (1) intragroup variability and interrater reliability of judges' estimates were statistically significant; (2) strengths of correlation coefficients for classroom test scores (CTS) and supervisory rating scores (SRS) under Angoff method exceeded r values for scores under conventional method; (3) strength of correlation coefficient for CTS and skill qualification test (SQT) scores under conventional method exceeded r value for scores under Angoff method; (4) correlation coefficients for CTS and SRS were statistically significant for Angoff "accepts" but not for Angoff "rejects" in three of four job performance areas, and means of SRS of the two groups of job performers were significantly different; and (5) correlation coefficient for CTS and SQT scores was statistically significant for Angoff "accepts" but not for Angoff "rejects" and means of SQT scores of the two groups of job performers were significantly different.;Conclusions. The Angoff cutoff score-setting method provides an effective means for setting criterion-referenced test cutoff scores. The Angoff and present score setting methods yield significantly different test standards. The score derived by the empirical method is a better measure of minimum job requirements of an entry-level performer, thereby enhancing the predictive validity of the classroom test. Recommendations for future research are included.
| Identifer | oai:union.ndltd.org:wm.edu/oai:scholarworks.wm.edu:etd-1521 |
| Date | 01 January 1988 |
| Creators | McDaniels, Darl |
| Publisher | W&M ScholarWorks |
| Source Sets | William and Mary |
| Language | English |
| Detected Language | English |
| Type | text |
| Format | application/pdf |
| Source | Dissertations, Theses, and Masters Projects |
| Rights | © The Author |
Page generated in 0.0013 seconds