published_or_final_version / Psychology / Doctoral / Doctor of Philosophy

published_or_final_version / Statistics / Master / Master of Philosophy

Keywords: Equating, IRT, Testlet Respons Model, Rasch Testlet Model, LSC, Concurrent, FPC / Test equating allows direct comparison of test scores from alternative forms measuring the same construct by employing equating procedures to put the test scores on the same metric. Three equating procedures are commonly used in the literature including the concurrent calibration method, the linking separate calibration methods (e.g. the moment methods and the characteristic curve methods), and FPC (Fixed Parameter Calibration) method. The first two types of methods for traditional IRT model have been well developed. FPC is being emphasized recently because of its utility for constructing item bank and computerized adaptive testing (CAT). However, there are few studies that examine the equating accuracy of the FPC method compared to that of the linking separate calibration method and the concurrent calibration method. / The equating methods for the traditional IRT model are not appropriate for linking testlet-based tests because the local independence assumption of IRT model cannot be held for this type of tests. Some measurement models, such as testlet response model, bi-factor model, and Rasch testlet model, were advanced to calibrate the models for the testlet-based tests. Few equating methods, however, that take into consideration the additional local dependence among the examinees' responses to items within testlets have been developed for linking testelet-based tests. / The first study compared the equating accuracies of the FPC, the linking separate calibration, and the concurrent calibration method based on the IRT model to equate item parameters under different conditions. The results indicated that the FPC method using BILOG-MG performed as well as the linking separate calibration method and the concurrent calibration method for linking the equivalent groups. However, the FPC method produced larger equating errors than the other two methods did when the ability distributions of the base and target groups were substantially nonequivalent. Differences in difficulties between the common items set and the total test did not substantially affect the equating results with the three methods, with other conditions being held equal. As expected, both small sample size and less number of common items led to slight greater equating errors. / The last study used the concurrent calibration method under the multidimensional Rasch testlet model to link the testlet-based tests in which the testlets were composed of dichotomous, polytomous, and mixed-format items. The results demonstrated that the concurrent calibration method under the Rasch testlet model worked well in recovering the underlying item parameters. Again, equating errors were substantially increased if the local dependences were ignored in model calibration. And smaller testlet variances for the common testlets led to more accurate equating results. / The results of the studies contribute to a better understanding of the effectiveness of the different equating methods, particularly those for linking testlet-based tests. They also help clarify influences of the other factors, such as characteristics of the examinees, features of the common items and common testlets on equating results. Testing practitioners and researchers may draw useful recommendations from the findings about equating method selection. Nevertheless, generalizations of the findings from the simulated studies to practical testing programs should be cautious. / The second study developed an item characteristic curve method and a testlet characteristic curve method for the testlet response model to transform the scale of item parameters. It then compared the effectiveness of the characteristic curve methods and the concurrent calibration methods under different conditions in linking item parameters from alternate test forms which were composed of dichotomously scored testlet-based items. The newly developed item characteristic curve method and the testlet characteristic curve method were shown to perform similarly as or even better than the Stocking-Lord test characteristic curve method and the concurrent calibration method did. Ignoring the local dependence in model calibration substantially increased equating errors. And larger testlet variances for the common testlets led to greater equating errors. / To address the need to better understand the FPC method and to develop new equating methods for linking testlet-based tests, the studies were to compare the effectiveness of the three types of equating methods under different linking situations and to develop equating methods for linking testlet-based tests. Besides the equating methods concerned, other factors, including sample size, ability distribution, and characteristics of common items and testlets that might affect equating results were also considered. Three simulation studies were carried out to accomplish the research purposes. / Zhang, Zhonghua. / Adviser: Yujing Ni. / Source: Dissertation Abstracts International, Volume: 72-01, Section: A, page: . / Thesis (Ph.D.)--Chinese University of Hong Kong, 2010. / Includes bibliographical references (leaves 156-166). / Electronic reproduction. Hong Kong : Chinese University of Hong Kong, [2012] System requirements: Adobe Acrobat Reader. Available via World Wide Web. / Electronic reproduction. Ann Arbor, MI : ProQuest Information and Learning Company, [200-] System requirements: Adobe Acrobat Reader. Available via World Wide Web. / Electronic reproduction. Ann Arbor, MI : ProQuest Information and Learning Company, [200-] System requirements: Adobe Acrobat Reader. Available via World Wide Web. / Abstract also in Chinese.

Determining which students are at-risk of poorer outcomes -- such as dropping out, failing classes, or decreasing standardized examination scores -- has become an important area of both research and practice in K-12 education. The models produced from this type of predictive modeling research are increasingly used by high schools in Early Warning Systems to identify which students are at risk and intervene to support better outcomes. It has become common practice to re-build and validate these detectors, district-by-district, due to different data semantics and various risk factors for students in different districts. As these detectors become more widely used, however, a new challenge emerges in applying these detectors across a broad spectrum of school districts with varying availability of past student data. Some districts have insufficient high-quality past data for building an effective detector. Novel approaches that can address the complex data challenges a new district presents are critical for advancing the field. Using an ensemble-based algorithm, I develop a modeling approach that can generate a useful model for a previously unseen district. During the ensembling process, my approach, District Similarity Ensemble Extrapolation (DSEE), weights districts that are more similar to the Target district more strongly during ensembling than less similar districts. Using this approach, I can predict student-at-risk status effectively for unseen districts, across a range of grade ranges, and achieve prediction goodness but ultimately fails to perform better than the previously published Knowles (2015) and Bowers (2012) EWS models proposed for use across districts.

published_or_final_version / Education / Master / Master of Education

It is hypothesized that useful diagnostic information can reside in the wrong answers of multiple-choice tests, and that properly designed distractors can yield indications of misinformation and missing information in algorithmic reasoning on the part of the test taker. In addition to summarizing the literature regarding diagnostic research as opposed to scoring research, this study proposes a methodology for analyzing test results and compares the findings with those from the research of Birenbaum and Tatsuoka and others. The proposed method identifies the conditions of misinformation and missing information, and it contains a statistical compensation for careless errors. Strengths and weaknesses of the method are explored, and suggestions for further research are offered.

This dissertation contributes to the ongoing debate on how educational inputs make a difference and how to allocate them efficiently. Educational inputs could be broadly defined as any personnel inputs such as teachers and career service staff, learning environment that includes peers and school facilities, and policies that facilitate learning. This dissertation explores three topics: peer effects in higher education, the consequences of college expansion, and the impacts of school closures. Chapter I estimates the peer effects of non-cognitive skills. I show how peers’ non-cognitive skills influence students' academic outcomes and own non-cognitive skills. I use a unique dataset that includes information on student non-cognitive skills, course grades, and friendship from a university in China that randomly assigns students to dormitories. My first main finding is that peers’ non-cognitive skills affect students’ academic outcomes positively but differentially. All students benefit from exposure to “persistent” peers, while students with low baseline academic ability also benefit from exposure to “motivated” peers. My second main finding is that peers also affect the development of students’ self-control and willingness to socialize. These findings have important implications in evaluating the social returns to interventions that improve non-cognitive skills and education policies that change peer group composition. Chapter II summarizes the current literature on college expansions, which change the education resource for many students. Studies have explored the impact of College Expansions that happened worldwide and this chapter summarizes literature in the field of economics of education. This chapter pays special attention to studies that explore the impact on wages and employment and how current studies identify causal relationships. Meanwhile, this chapter reviews how current studies examine the impacts of college expansion in China starting from 1999, which was unparalleled in magnitude. Finally, I discuss how future studies could improve to identify causal effects of the impact of the tremendous college expansion in China. Chapter III, a joint work with Ying Xu, estimates the effect of school closures causedby wildfires. School closures are a common and disruptive feature of education systems when sudden shocks from weather, natural disasters, or infectious disease require that students remain at home rather than in the classroom. Indeed, since January 2020, school closures have happened all around the world due to the COVID-19 pandemic. In the United States, more than 50 million students are currently out of school due to COVID-related closures. This raises an important question: How do sudden school closures affect student development in the short and medium term? In this chapter, we use administrative data to examine the causal effect of unexpected school closures, exploiting sudden variations in these closures due to wildfires in California. We show that unexpected closures have negative effects on student test scores, and the loss of school time is one of the most important mechanisms of decline in student achievement. Meanwhile, minority students and students from school districts with low socioeconomic status experience larger negative effects from such unexpected closures. We argue that these results can help inform policy to identify and address the negative impacts of such closures.

This study looked at the environmental factors present during testing for the spring 2004 administration of the California High School Exit Exam (CAHSEE) in an attempt to quantify some of the factors that were previously only qualitatively reported. Five factors were examined for their ability to predict passing percentages of students on the CASHSEE at the school level. The results indicated that socioeconomic status was the only significant predictor.

published_or_final_version / Education / Master / Master of Education