University of North Texas
Automated Methods to Evaluate the Quality of Distractors in College-Level MCQS
Abstract
dc:descriptionGenerative artificial intelligence holds significant potential to transform the field of education, particularly by assisting researchers in creating course materials and assessment items. One critical component in the generation of multiple-choice questions (MCQs) is the construction of distractors or incorrect answer options. Effective distractors must maintain a delicate balance between plausibility, i.e., the ability to mislead students who lack full mastery of the subject, and clarity: to appear incorrect to those who understand the material. Despite advancements in automated question generation, there remains a notable gap in the formal evaluation of distractor quality. Specifically, little work has been done to define what constitutes a good distractor or to develop reliable methods for ranking multiple distractors. In this work, we propose several automated approaches for evaluating distractor quality, with the goal of identifying those that most closely align with human judgment. We begin by constructing a dataset in which human annotators rank distractors post hoc, using questions sampled from established multiple-choice question (MCQ) benchmarks such as ARC and MMLU. We then evaluate various prompting-based methods to assess the extent to which state-of-the-art large language models (LLMs) correlate with human rankings. Finally, we introduce a novel technique, "SLMs as Examinees" wherein multiple small language models (SLMs), trained to reflect varying levels of subject expertise, simulate human test-takers. Distractor rankings are then derived from the performance of these models. This work thus provides a foundation for the development of more principled and effective distractor generation systems.
Degree
thesis:*- Grantor dc:publisher
- University of North Texas
- Year dc:date
- 2025
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Vegi, Deekshitth 2000-
- Contributors dc:contributor
-
- Pears, Russel
- Shah, Sayed K
- Ray Choudhury, Sagnik
Subjects
dc:subject × 3Rights
dc:rights- Statement dc:rights
-
- Public
- Vegi, Deekshitth 2000-
- Copyright
- Copyright is held by the author, unless otherwise noted. All rights Reserved.
- Language dc:language
- English
Identifiers
dc:identifier.*- Identifier
-
local-cont-no: submission_4552
ark: ark:/67531/metadc2566435