Policymakers expect international educational assessments to report credible national and international changes in student achievement over time. However, international assessment projects face great methodological challenges to creating comparable scores across jurisdictions and time points, fundamentally because jurisdictions vary in many aspects of curriculum and curriculum change as well as in the patterns of students’ test-taking behaviour. Using data from the Second IEA Mathematics Study (SIMS), the study reported in this dissertation addresses the potential impact of the different equating methodologies used in current international assessments on the accurate estimates of change in jurisdiction achievement over time. The results of the study demonstrate that the different equating methodologies implemented through the Item Response Theory (IRT) models currently used in international assessments may be of limited use in estimating change in jurisdiction achievement over time. This is because the international assessment data violate the IRT model assumptions, in particular the unidimensionality assumption. In addition, the estimation of jurisdiction results based on a common international scale may potentially distort the results of those jurisdictions that have levels of student achievement that are much lower or higher than most other participating jurisdictions. The findings of this study have important implications for researchers as well as policy makers.

Policymakers expect international educational assessments to report credible national and international changes in student achievement over time. However, international assessment projects face great methodological challenges to creating comparable scores across jurisdictions and time points, fundamentally because jurisdictions vary in many aspects of curriculum and curriculum change as well as in the patterns of students’ test-taking behaviour. Using data from the Second IEA Mathematics Study (SIMS), the study reported in this dissertation addresses the potential impact of the different equating methodologies used in current international assessments on the accurate estimates of change in jurisdiction achievement over time. The results of the study demonstrate that the different equating methodologies implemented through the Item Response Theory (IRT) models currently used in international assessments may be of limited use in estimating change in jurisdiction achievement over time. This is because the international assessment data violate the IRT model assumptions, in particular the unidimensionality assumption. In addition, the estimation of jurisdiction results based on a common international scale may potentially distort the results of those jurisdictions that have levels of student achievement that are much lower or higher than most other participating jurisdictions. The findings of this study have important implications for researchers as well as policy makers.

When applying cognitive diagnostic models, the goal is to accurately estimate students’ diagnostic profiles. The accuracy of these estimates may be enhanced by looking at the types of incorrect options a student selects. This thesis research examines the additional diagnostic information available from the distractors in multiple-choice items used in large-scale achievement assessments and identifies optimal conditions for extracting diagnostic information. The study is based on the analyses of both real student responses and simulated data. The real student responses are from a large-scale provincial math assessment for grade 6 students in Ontario. Data were then simulated under different skill dimensionality and item discrimination conditions. Comparisons were made between student profile estimates when using the DINA and MC-DINA models. The MC-DINA model is a newly developed cognitive diagnostic model where the probability of a student choosing a particular item option depends on how closely the student’s cognitive skill profile matches the skills tapped by that option. The results from the simulation data analysis suggested that when the simulated data included additional diagnostic information in the distractors, the MC-DINA model was able to use that information to improve the estimation of the student profiles, which shows the utility of the additional information obtained from item distractors. The value of adding information from distractors was greater when there was lower item discrimination and more skill multidimensionality. However, in the real data, the keyed options provided more diagnostic information than the distractors, and there was little information in the distractors that could be utilized by the MC-DINA model. This implies that current math test items could be further developed to include diagnostically rich distractors. The study offers some suggestions for a design of multiple-choice test items and its formative use.

When applying cognitive diagnostic models, the goal is to accurately estimate students’ diagnostic profiles. The accuracy of these estimates may be enhanced by looking at the types of incorrect options a student selects. This thesis research examines the additional diagnostic information available from the distractors in multiple-choice items used in large-scale achievement assessments and identifies optimal conditions for extracting diagnostic information. The study is based on the analyses of both real student responses and simulated data. The real student responses are from a large-scale provincial math assessment for grade 6 students in Ontario. Data were then simulated under different skill dimensionality and item discrimination conditions. Comparisons were made between student profile estimates when using the DINA and MC-DINA models. The MC-DINA model is a newly developed cognitive diagnostic model where the probability of a student choosing a particular item option depends on how closely the student’s cognitive skill profile matches the skills tapped by that option. The results from the simulation data analysis suggested that when the simulated data included additional diagnostic information in the distractors, the MC-DINA model was able to use that information to improve the estimation of the student profiles, which shows the utility of the additional information obtained from item distractors. The value of adding information from distractors was greater when there was lower item discrimination and more skill multidimensionality. However, in the real data, the keyed options provided more diagnostic information than the distractors, and there was little information in the distractors that could be utilized by the MC-DINA model. This implies that current math test items could be further developed to include diagnostically rich distractors. The study offers some suggestions for a design of multiple-choice test items and its formative use.

Master of Science / Curriculum and Instruction Programs / Kay A. Taylor / This historical and qualitative inquiry investigates recent educational reform in Ukraine. On Tuesday, April 22, 2008 more than half a million Ukrainian high-school graduates were ordered to take an external assessment of student achievements. The new assessment model was named the External Independent Knowledge Testing and replaced the traditional forms of high school exit-entrance to higher educational institution exams in Ukraine. These changes in assessment strategies in the Ukrainian educational system illustrate the global trend of replacing diverse forms of national examinations with standardize multiple-choice assessments that can be scored by machines and could be viewed as a new page in the history of educational assessment. Educational assessment has become one of the most significant areas of research in the United States. It is not only a prominent issue in education today but also one of the most controversial issues in contemporary educational science. The debate is concentrated around the question of what purpose educational assessment serves. A growing body of international research suggests that educational assessment is a new mechanism of social and political control aimed to legitimize social inequalities. Thus, the primary goal of this study was to investigate what kind of changes the External Independent Knowledge Testing in Ukraine was expected to bring to Ukrainian society and whether the initial results of the reform were consistent with what was expected. Data was collected by means of interview, survey, and examining Ukrainian publications. The data revealed that the recent changes in traditional practices of assessment in Ukraine were aimed at serving much broader social purposes than those identified by Ukrainian policymakers. Under the guise of improving the quality of Ukrainian education and fighting corruption in higher education, the educational reforms in Ukraine are implementing an undetected new form of social and political preferences through education and testing. Thus, the reform helps new Ukrainian nobilities legitimize their status through the new system of exercising control over Ukrainian education.

This study is an investigation of the perspectives and practices of U.K. expert practitioners regarding the accessibility of tests in higher education online learning environments. As a growing area of concern in the higher education field, the accessibility of online tests is a particularly complex and challenging topic. To explore this topic further, twelve U.K. higher education expert practitioners were consulted. Five major themes that were identified by the participants included: 1. Requirement for balance between academic integrity and accessibility; 2. Need for inclusive design to better support the creation and delivery of accessible online tests; 3. Issues related to the mainstreaming of accessible online tests and testing processes; 4. Resources required to support the design and delivery of accessible online tests; and 5. Technology issues and approaches. The participants shared their research findings, experiences, and reflections regarding their work related to different types of online tests and accessibility.

This study is an investigation of the perspectives and practices of U.K. expert practitioners regarding the accessibility of tests in higher education online learning environments. As a growing area of concern in the higher education field, the accessibility of online tests is a particularly complex and challenging topic. To explore this topic further, twelve U.K. higher education expert practitioners were consulted. Five major themes that were identified by the participants included: 1. Requirement for balance between academic integrity and accessibility; 2. Need for inclusive design to better support the creation and delivery of accessible online tests; 3. Issues related to the mainstreaming of accessible online tests and testing processes; 4. Resources required to support the design and delivery of accessible online tests; and 5. Technology issues and approaches. The participants shared their research findings, experiences, and reflections regarding their work related to different types of online tests and accessibility.

Doctor of Philosophy / Department of Special Education, Counseling and Student Affairs / Fred O. Bradley / Over the last few years, the safety record of U.S. commercial airlines has improved to the point where the statistics on accidents are negligible. The overwhelming numbers of aviation fatalities occur in General Aviation (GA) accidents. While the fatal accident rate has improved—from around 5.0 per 100,000 miles flown in the post–World War II era to varying between 1.2 and 1.5 since 1996—it still results in 450 to 700 deaths per year. In 2013, improving GA safety was on the National Transportation Safety Board’s (NTSB’s) most wanted list. The NTSB has cited a lack of aeronautical knowledge as the cause of many of these accidents. If pilots are required to pass Federal Aviation Administration (FAA) knowledge and practical tests prior to obtaining a new pilot certificate, how could they not possess the knowledge needed to operate in the National Airspace System (NAS)? Some, attributing it to a failure to learn basic aeronautical knowledge, are concerned that potential pilots memorize the answers to test questions published in commercially available test guides and quickly forget the material after passing the test. The purpose of this retrospective causal comparative study was to see if airman knowledge tests scores are related to fatal accidents. Fatal GA accidents that had pilot error as a causal factor were compared with those in which maintenance factors caused the accidents, to see if there were significant differences in the pilots’ knowledge test scores. The time that potential pilots took to answer skill-based questions that required calculation or interpretation was compared to the time to answer recall questions to see if rote memorization may have been involved in passing the knowledge test. The results of this paper may have implications on how AFS 630 structures the FAA knowledge tests and how instructors prepare potential pilots for these tests.

Doctor of Philosophy / Curriculum and Instruction Programs / Andrew G. Bennett / Online assessments afford many advantages for teachers and students. Okolo (2006) stated, “As the power, sophistication, and availability of technology have increased in the classroom, online assessments have become a viable tool for providing the type of frequent and dynamic assessment information that educators need to guide instructional decisions,” (pp 67-68). As post secondary institutes use online learning environments, education has molded into hybrid experiences. Traditional courses now regularly infuse components of online learning and assessments by required student participation both in person and online. Research is needed to analyze online components of assessment and student achievement. Data was gathered from an undergraduate mathematics course designed for students seeking a bachelor’s degree in elementary education. The course was entitled MATH 320: Mathematics for Elementary School Teachers. Synergies of quantitative and qualitative data were evaluated to assess the impact of written and video help tutorials in online quizzes on student achievement. Three forms of data were collected: student interviews, surveys about students’ online quiz experiences and learning style preferences, and student performance and tutorial usage statistics from seven online quizzes. Student interviews were conducted mid-semester by the researcher who also transcribed and analyzed data. Graphical schemes were used to identify and categorize responses to interview questions. Students’ responses were summarized and quantified in frequency tables. Surveys about students’ online quiz experiences and learning style preferences were analyzed through descriptive statistical methods to describe the data with numerical indices and in graphical form. Correlation matrices and linear regression models were used to identify relationships among survey items. Additionally, Analysis of Variance (ANOVA) techniques were used to explore the data for statistical significance. Students were assigned seven online quizzes throughout the semester. Descriptive statistics were calculated to describe the online quiz data. Regression models were used to determine correlations between use of help tutorials and performance on online quizzes. Data analysis revealed students were persistent and motivated to retake similar quizzes multiple times until a high or perfect score was obtained. After missing a problem, students selected written help tutorials more often than video help tutorials to identify mistakes and understand how to solve the particular problem. The proportion of students whose scores improved after using both written and video help tutorials was greater than those who used the written help tutorials alone. Although the number of students who benefited from the video help tutorials was smaller than expected, the increased performance could be appreciated by students and educators alike. The research presented herein should serve as a base for curriculum development in university mathematics programs utilizing or considering implementation of online tutorials coupled with student evaluation.

This study built and supported arguments for the use of diagnostic assessment in English as a second language (ESL) academic writing. In the two-phase study, a new diagnostic assessment scheme, called the Empirically-derived Descriptor-based Diagnostic (EDD) checklist, was developed and validated for use in small-scale classroom assessment. The checklist assesses ESL academic writing ability using empirically-derived evaluation criteria and estimates skill parameters in a way that overcomes the problems associated with the number of items in diagnostic models. Interpretations of and uses for the EDD checklist were validated using five assumptions: (a) that the empirically-derived diagnostic descriptors that make up the EDD checklist are relevant to the construct of ESL academic writing; (b) that the scores derived from the EDD checklist are generalizable across different teachers and essay prompts; (c) that performance on the EDD checklist is related to performance on other measures of ESL academic writing; (d) that the EDD checklist provides a useful diagnostic skill profile for ESL academic writing; and (e) that the EDD checklist helps teachers make appropriate diagnostic decisions and has the potential to positively impact teaching and learning ESL academic writing. Using a mixed-methods research design, four ESL writing experts created the EDD checklist from 35 descriptors of ESL academic writing. These descriptors had been elicited from nine ESL teachers’ think-aloud verbal protocols, in which they provided diagnostic feedback on ESL essays. Ten ESL teachers utilized the checklist to assess 480 ESL essays and were interviewed about its usefulness. Content reviews from ESL writing experts and statistical dimensionality analyses determined that the underlying structure of the EDD checklist consists of five distinct writing skills: content fulfillment, organizational effectiveness, grammatical knowledge, vocabulary use, and mechanics. The Reduced Reparameterized Unified Model (Hartz, Roussos, & Stout, 2002) then demonstrated the diagnostic quality of the checklist and produced fine-grained writing skill profiles for individual students. Overall teacher evaluation further justified the validity claims for the use of the checklist. The pedagogical implications of the use of diagnostic assessment in ESL academic writing were discussed, as were the contributions that it would make to the theory and practice of second language writing instruction and assessment.