Back to search

University of Edinburgh

Exploring examiner variability in undergraduate OSCEs: hawks, doves and those in between

Abstract

dc:description.abstract

OSCEs are used as a major part of undergraduate /primary medical degree summative assessments, and involve judgements by examiners on individual candidates’ performance. Medical schools have a duty of care to the general public, as well as to their students, to have assessments which are able to delineate between candidates who do or do not meet the required standard. Errors in assessment may lead to candidates who are not competent becoming doctors, which is unsafe, or to candidates who are competent not becoming doctors, which is both unfair on those candidates and wasteful of educational resources. Examiner behaviour may influence how likely students are to pass their OSCE, and so institutions should try to explore this, which is challenging given the small sizes of medical school candidate and examiner cohorts. From 2024 the General Medical Council required that all UK medical degrees incorporate the Medical Licensing Assessment (MLA), which includes requirements to describe and demonstrate marking calibration. This project aimed to explore a pragmatic, potentially reproducible method for examining examiner behaviours within a university-sized cohort, to see whether examiner factors such as gender, ethnicity, age and experience influence how they mark, and to what extent this impacts candidates’ likelihood of passing or failing the assessment. The open-source software R was used for data analysis. When comparing different examiner groups, I found that there were no major differences in examiner variability between demographic groups (gender, ethnicity, age), although analyses for ethnicity were limited by group sizes. More senior clinicians tended to mark across a broader range, and marginally more leniently. However, in general the examiners with (proxy markers for) more clinical experience did not exhibit different behaviours to those with less experience. There was minimal difference in marking behaviours between examiners with differing levels of experience at examining OSCEs, and recency of examiner training did not appear to significantly influence examiner behaviour. This project demonstrated the feasibility of this method of analysing local data within a medical school, by a clinician rather than a psychometrician or statistician – a pragmatic approach to studying variability within the small examiner cohorts common in undergraduate OSCEs and a practical method of meeting the GMC MLA requirements, which can be undertaken by clinician-educators with no data science or statistics training. Further, the R code developed in this project could be used by other medical schools, streamlining the process by removing the lengthy development stages.

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Layden, Elizabeth Anne
Advisors dc:contributor.advisor
  • Japp, Alan
  • Hope, David

Subjects

dc:subject × 9

Rights

Language dc:language.iso
en

Identifiers

dc:identifier.*

Chain of custody

source
Harvested from
University of Edinburgh
Base URL
era.ed.ac.uk/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Layden, Elizabeth Anne. Exploring examiner variability in undergraduate OSCEs: hawks, doves and those in between. 2026. https://era.ed.ac.uk/handle/1842/44920