This research examines an ordered perception rater model, an extension of the equal perception signal detection theory (SDT) latent class rater model. The expectation-maximization algorithm and the Newton-Raphson algorithm are used to estimate parameters. Four simulation studies are conducted to answer three research questions.
Simulation studies 1 and 2 fit correct models to the data. Simulation study 1 generates one hundred data sets from the equal perception rater model, both with fully-crossed design and BIB design, and both without and with rater effects, and fits the equal perception model. Parameter recovery is excellent for fully-crossed design and reasonable for BIB design, and all rater effects are detected. Simulation study 2 generates one hundred simulated data sets from the ordered perception model, both with fully-crossed design and BIB design, and both without and with rater effects, and fits the ordered perception rater model. Although parameter recovery is biased for some parameters in the BIB design, all rater effects are recovered.
Simulation studies 3 and 4 fit wrong models to the data. Simulation study 3 fits equal perception models to the fully-crossed and BIB ordered perception data sets generated in simulation study 2. All rater effects are revealed, although rater effects are distorted to some extent in the BIB design. Simulation study 4 fits ordered perception models to the fully-crossed and BIB equal perception data sets generated in study 1. All rater effects are recovered.
Using essay scores from a large-scale language test, an empirical study is conducted. Both the equal and the ordered perception models are fitted. Information criteria favor the equal perception model.
| Identifer | oai:union.ndltd.org:columbia.edu/oai:academiccommons.columbia.edu:10.7916/d8-s55z-sv06 |
| Date | January 2019 |
| Creators | Zhou, Xiaoliang |
| Source Sets | Columbia University |
| Language | English |
| Detected Language | English |
| Type | Theses |
Page generated in 0.002 seconds