Most in-hospital adverse events are attributable to surgical care and of these, clinical decision-making (CDM) errors account for approximately half. The assessment of CDM is integral to establishing competence among surgical trainees. One proposed assessment tool is the script concordance test (SCT), which is based on the dual process theory of CDM, but evidence demonstrating valid results is needed. This thesis collects content and response process validity evidence for the assessment of CDM using the SCT.
To gather content evidence, a Delphi technique was conducted with a panel of local general surgeons (n=15) consisting of the top decision-makers voted by staff and resident surgeons. Items achieving consensus were mapped on a table of specifications to determine the breadth of topics covered as defined by the Royal College medical expert competencies in general surgery. The final SCT was administered to 29 residents and 14 staff surgeons and results were analyzed. To gather response process evidence, cognitive interviews were then conducted with ten residents and five staff surgeons based on results of the final SCT. Data from the cognitive interviews were analyzed using a priori deductive codes based on Tourangeau’s cognitive model of response process.
The first round of the Delphi yielded agreement ranging from 40-100% and consensus for 21 cases. The 21 cases made up the final SCT and encompassed 13 of the 19 competencies in general surgery. The final SCT reflected a test of the intraoperative management of open, adult general surgery. Notable absent competencies were described by experts to be outside the scope of general surgery, too difficult for the resident level of training, or presenting an unrealistic intraoperative finding.
Cognitive interviews demonstrated variability in CDM among test-takers. Consistent with the dual process theory, test-takers relied on scripts formed through past experiences, when available, to make decisions. However, test-takers’ response process was also influenced by issues with respect to their comprehension, recall, and response matching cognitive steps. Due to issues with response matching in particular, when answering an SCT question test-takers indicating different numerical ratings may have the same rationale.
The Delphi technique, table of specifications, and cognitive interviews provide validity evidence supporting the SCT for assessing CDM of general surgery trainees. Substantial issues with respect to the numerical rating scale suggests further revisions to the test format are required before consideration of its use in summative assessment.
| Identifer | oai:union.ndltd.org:uottawa.ca/oai:ruor.uottawa.ca:10393/38383 |
| Date | 02 November 2018 |
| Creators | Gawad, Nada |
| Contributors | Wood, Timothy |
| Publisher | Université d'Ottawa / University of Ottawa |
| Source Sets | Université d’Ottawa |
| Language | English |
| Detected Language | English |
| Type | Thesis |
| Format | application/pdf |
Page generated in 0.0021 seconds