Back to search

University of North Texas

Automated Methods to Evaluate the Quality of Distractors in College-Level MCQS

Abstract

dc:description

Generative artificial intelligence holds significant potential to transform the field of education, particularly by assisting researchers in creating course materials and assessment items. One critical component in the generation of multiple-choice questions (MCQs) is the construction of distractors or incorrect answer options. Effective distractors must maintain a delicate balance between plausibility, i.e., the ability to mislead students who lack full mastery of the subject, and clarity: to appear incorrect to those who understand the material. Despite advancements in automated question generation, there remains a notable gap in the formal evaluation of distractor quality. Specifically, little work has been done to define what constitutes a good distractor or to develop reliable methods for ranking multiple distractors. In this work, we propose several automated approaches for evaluating distractor quality, with the goal of identifying those that most closely align with human judgment. We begin by constructing a dataset in which human annotators rank distractors post hoc, using questions sampled from established multiple-choice question (MCQ) benchmarks such as ARC and MMLU. We then evaluate various prompting-based methods to assess the extent to which state-of-the-art large language models (LLMs) correlate with human rankings. Finally, we introduce a novel technique, "SLMs as Examinees" wherein multiple small language models (SLMs), trained to reflect varying levels of subject expertise, simulate human test-takers. Distractor rankings are then derived from the performance of these models. This work thus provides a foundation for the development of more principled and effective distractor generation systems.

Degree

thesis:*
Grantor dc:publisher
University of North Texas
Year dc:date
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Vegi, Deekshitth 2000-
Contributors dc:contributor
  • Pears, Russel
  • Shah, Sayed K
  • Ray Choudhury, Sagnik

Subjects

dc:subject × 3

Rights

dc:rights
Statement dc:rights
  • Public
  • Vegi, Deekshitth 2000-
  • Copyright
  • Copyright is held by the author, unless otherwise noted. All rights Reserved.
Language dc:language
English

Identifiers

dc:identifier.*
Identifier
local-cont-no: submission_4552
ark: ark:/67531/metadc2566435

Chain of custody

source
Harvested from
University of North Texas
Base URL
digital.library.unt.edu/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Vegi, Deekshitth 2000-. Automated Methods to Evaluate the Quality of Distractors in College-Level MCQS. University of North Texas, 2025. https://digital.library.unt.edu/ark:/67531/metadc2566435/