Background: Providing examinee performance data is a common procedural modification in modified-Angoff standard setting exercises. Results of research examining the impact of the practice are mixed, and despite the frequency with which this procedural modification is done, the overall impact on the outcomes is not well understood. This research presents a large-scale evaluation of the impact of providing performance data in the context of operational standard setting exercises for a medical licensing examination program. Methods: Data from a total of 18 independent standard setting panels across three different examinations were analyzed to evaluate whether and how the provision of performance information impacted the resulting cut scores. Results: Significant cut-score changes were found between initial and final judgments for all panels. A decrease in standard deviation between initial and final judgments for all panels was indicative of the convergence of judgments following the provision of data, and a lack of systematic changes in final cut scores suggested that judges were not simply changing their content-based judgments to align with the normative data. Both findings provide important procedural validity evidence for the standard setting process. Implications: The current data set (replications of the procedure within a single cycle, across cycles for an individual examination, and across multiple examinations) allowed for the most comprehensive evaluation of this topic that has been described to date and provides valuable insight into an issue that is relevant to many testing contexts. / Educational Psychology
| Identifer | oai:union.ndltd.org:TEMPLE/oai:scholarshare.temple.edu:20.500.12613/1832 |
| Date | January 2011 |
| Creators | Margolis, Melissa J. |
| Contributors | DuCette, Joseph P., Farley, Frank, Fullard, William, Thurman, S. Kenneth, Clauser, Brian E. |
| Publisher | Temple University. Libraries |
| Source Sets | Temple University |
| Language | English |
| Detected Language | English |
| Type | Thesis/Dissertation, Text |
| Format | 124 pages |
| Rights | IN COPYRIGHT- This Rights Statement can be used for an Item that is in copyright. Using this statement implies that the organization making this Item available has determined that the Item is in copyright and either is the rights-holder, has obtained permission from the rights-holder(s) to make their Work(s) available, or makes the Item available under an exception or limitation to copyright (including Fair Use) that entitles it to make the Item available., http://rightsstatements.org/vocab/InC/1.0/ |
| Relation | http://dx.doi.org/10.34944/dspace/1814, Theses and Dissertations |
Page generated in 0.0015 seconds