Back to search

University of North Dakota

Impact Of Delphi-Developed Rubrics On Rater Consistency In 14 CFR 141 Flight Training Assessment Environments

Abstract

dc:description.abstract

Pilot schools operating in accordance with 14 CFR 141 provide a substantial supply of pilots to the aviation industry. Recent high industry demand for pilots has created turnover among the flight instructor workforce, often considered an entry-level position used by aspiring aviators to build flight experience. This in turn means that check instructors, evaluators who are often selected from the more senior flight instructor population at a pilot school, are generally less experienced than in the past. Anecdotal evidence suggests that there is low consistency among these check instructors when conducting performance-based assessments. This study sought to determine the existing level of consistency among check instructors at a pilot school, explore the use of the Delphi method to create descriptive rubrics to be used to assess pilots on performance-based assessments, and evaluate the effectiveness of these rubrics in improving rater consistency among these check instructors. Results were mixed, indicating improvement in rater consistency in some tasks, but degraded consistency in others. The Delphi method proved to be an efficient means to gather consensus and establish written descriptions for a performance-based rubric. The methods utilized in this study provide a simple means for flight training providers and regulators to assess existing levels of consistency among check instructors and develop assessment instruments intended to enhance assessment validity and reliability.

Degree

thesis:*
Name thesis:degree_name
Doctor of Education (EdD)
Level thesis:degree_level
Dissertation
Discipline thesis:degree_discipline
Educational Leadership

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Archer, Lewis Robert James
Contributors dc:contributor
  • Woei Hung

Subjects

dc:subject × 1

Identifiers

dc:identifier.*
Repository record dc:identifier
https://commons.und.edu/theses/6518

Chain of custody

source
Harvested from
University of North Dakota
Base URL
commons.und.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Archer, Lewis Robert James. Impact Of Delphi-Developed Rubrics On Rater Consistency In 14 CFR 141 Flight Training Assessment Environments. Dissertation thesis, https://commons.und.edu/theses/6518