The purpose of the study was to investigate the impact of several testing variables—level of item quality, item bank size, placement of passing score, and computer-based test design, on two important indicators of the quality of credentialing exams, decision consistency and decision accuracy. The computer-based test designs of interest in this study were linear parallel form tests, multi-stage tests, and computerized adaptive tests. The choice of test design model plays an important role in both minimizing decision-making error and increasing test efficiency. However, of equal, if not greater, importance are the resources available to test developers: the number of items in the bank, and the quality of the items. Simulation studies were conducted to investigate the effects of these testing variables on the accuracy and consistency of binary decision-making of credentialing exams. These variables are all very essential and manipulable in practice and therefore their roles are especially worthy of investigation. To the extent possible, realistic situations were simulated to increase the generalizability of the findings. The main findings from the study were as follows: (1) improvements in item quality had the desirable effect of increasing decision accuracy and decision consistency by a practically significant amount; (2) doubling bank size helped significantly in lowering item exposure but had little direct impact on decision accuracy and decision consistency; (3) the location of the passing score significantly impacted on test length necessary to achieve desirable levels of test statistics, and (4) all three computer-based test designs produced comparable results in the conditions simulated. Two main conclusions can be drawn from the findings. First, steps to increase item quality such as improving item writer training, cloning the best items, and improving and extending field testing of new items, offer the potential for improving the statistical characteristics of pass-fail credential exam decisions. Second, the general attractiveness of new computer-based test designs may be less important when pass-fail decisions rather than precise ability estimates are the focus of the examination. Clearly, more research to pursue these two conclusions, and confirm them, if possible, is in order.
| Identifer | oai:union.ndltd.org:UMASS/oai:scholarworks.umass.edu:dissertations-3535 |
| Date | 01 January 2001 |
| Creators | Xing, Dehui |
| Publisher | ScholarWorks@UMass Amherst |
| Source Sets | University of Massachusetts, Amherst |
| Language | English |
| Detected Language | English |
| Type | text |
| Source | Doctoral Dissertations Available from Proquest |
Page generated in 0.0016 seconds