The purpose of this study was to investigate the effect of guessing in the assessment of dimensionality in multiple-choice tests using procedures implemented in Mplus and TESTFACT. Levels of item discrimination and the size of correlations between dimensions were also manipulated to explore any interaction between these effects. Four indices based on the proportion of variance, parallel analysis, RMSR reduction and a chi-square difference test were used to estimate dimensionality. The research included two parts, a simulation study using a Monte Carlo approach and an application with TIMSS 2003 data.
The simulation study confirmed the guessing effect. TESTFACT appeared to outperform Mplus for most conditions with data that assumed guessing. The proportion of variance and the RMSR reduction indices more accurately estimated dimensionality in Mplus, whereas the chi-square test and parallel analysis performed best with TESTFACT. A discrimination effect was observed clearly in data that assumed no guessing using the parallel analysis index and in data that assumed guessing using the RMSR index for both methods. Less accurate estimation of dimensionality was observed when using Mplus for tests with either high or low discriminating items, and with TESTFACT for tests with lower discriminating items. Higher correlations between dimensions led to more serious estimation problems. When guessing was not modeled, greater influence from the levels in correlations between dimensions and item discriminations was found. Further, a more pronounced discrimination effect was observed in the high correlation condition.
With regard to the application of TIMSS data, 70% of the items exhibited guessing behaviors and high correlations were observed between scores on the different dimensions (math and science). Based on the simulation study, guessing and correlation effects should thus be considered carefully when choosing a method for assessing dimensionality. Inconsistency in the dimensionality assessment using the four indices with Mplus was observed (1 to 5 dimensions), whereas TESTFACT consistently estimated 2 dimensions. However, further investigation of the internal structure of the TIMSS assessment did not show any connection to content or cognitive domains.
| Identifer | oai:union.ndltd.org:PITT/oai:PITTETD:etd-04182007-060503 |
| Date | 27 June 2007 |
| Creators | Yeh, Chien-Chi |
| Contributors | Levent Kirisci, Suzanne Lane, Kevin H. Kim, Clement A. Stone |
| Publisher | University of Pittsburgh |
| Source Sets | University of Pittsburgh |
| Language | English |
| Detected Language | English |
| Type | text |
| Format | application/pdf |
| Source | http://etd.library.pitt.edu/ETD/available/etd-04182007-060503/ |
| Rights | unrestricted, I hereby certify that, if appropriate, I have obtained and attached hereto a written permission statement from the owner(s) of each third party copyrighted matter to be included in my thesis, dissertation, or project report, allowing distribution as specified below. I certify that the version I submitted is the same as that approved by my advisory committee. I hereby grant to University of Pittsburgh or its agents the non-exclusive license to archive and make accessible, under the conditions specified below, my thesis, dissertation, or project report in whole or in part in all forms of media, now or hereafter known. I retain all other ownership rights to the copyright of the thesis, dissertation or project report. I also retain the right to use in future works (such as articles or books) all or part of this thesis, dissertation, or project report. |
Page generated in 0.0022 seconds