Back to search

McMaster University

Automating Outcome Prioritization in Guideline Development: An artificial intelligence validation study

Abstract

dc:description.abstract

Background: Health outcome identification and prioritization are important steps for developing health guidelines that reflect the perspective of the target end-users. Research is needed on the methodology for prioritizing health guidelines. With the increasing integration of artificial intelligence (AI) tools into the evidence synthesis process, AI tools have the potential to aid in the other steps of the guideline development process beyond evidence synthesis. While guideline development groups have started to use AI for the identification of priority health outcomes, there are currently no studies examining the use and validation of AI tools during the outcome prioritization stage of the guideline development process. Methods: We conducted an AI validation study to investigate the accuracy and reproducibility of outcome identification and prioritization steps completed by custom GPTs created on ChatGPT, Google Gemini, and Microsoft Copilot models named “Outcome Prioritization for Importance (OPI)” to identify and prioritize relative importance of health outcomes compared to human guideline panels and untrained GPTs. Prompts were run at multiple timepoints. Jaccard Index Scores, Cohen’s Kappa, and Fleiss’ Kappa were used to measure similarity and agreement between the GPTs and humans, with Jaccard Index Scores and Kappa values of 0.5 and 0.61 as the cut-off indicators of concordance, respectively. Jaccard Index Scores and Fleiss’ Kappa were also used to assess reproducibility. Percent agreement was used to determine agreement between outcome importance ratings for prioritization. Time spent running prompts through GPTs was summarized narratively. Results: A total of 3501 outcomes were prioritized among the GPTs and human guideline panels. Gemini-OPI and ChatGPT-OPI had the highest mean Jaccard scores during internal and external validation, but none of the GPTs had a mean Jaccard score above 0.5. Cohen’s kappa values indicated there was no agreement between the custom GPTs versus humans and the untrained GPTs versus humans. ChatGPT-OPI had the highest similarity per recommendation question across all days, and a majority of the outcome lists generated per recommendation question by the GPTs had moderate-to-high similarity across all days. None of the GPTs had substantial or high agreement across all 3 days for outcome prioritization. Two hundred and ninety-two out of 421 (69.3%) of the outcome importance ratings had 100% agreement per recommendation question for all GPTs. Time spent running prompts through GPTs varied depending on optimization of workflow. Conclusion: This AI validation study used validation methods to assess outputs generated by customized GPT models built in ChatGPT, Gemini, and Copilot to human guideline panel members and untrained GPTs for the purpose of identifying and prioritizing health outcomes that are most important for decision-making. The results of this study can be used to refine the existing custom GPTs, design additional studies to assess model performance, and apply techniques to other validation studies for LLMs in the guideline development process.

Degree

thesis:*
Department dc:contributor.department
Health Research Methodology
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Green-Campbell, Dana
Advisor dc:contributor.advisor
  • Nieuwlaat, Robby

Rights

Language dc:language.iso
en

Identifiers

dc:identifier.*

Chain of custody

source
Harvested from
McMaster University
Base URL
macsphere.mcmaster.ca/server/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
related terms
citation

Green-Campbell, Dana. Automating Outcome Prioritization in Guideline Development: An artificial intelligence validation study. 2026. https://hdl.handle.net/11375/33876