Back to search

Boston University

Systematic analysis of large language model descriptor classification in pulmonary embolism

Abstract

dc:description.abstract

Pulmonary embolism (PE) remains the third leading cause of cardiovascular mortality, affecting over 600,000 Americans annually. Large language models (LLMs) have demonstrated promising performance for binary PE detection from radiology reports, yet their capacity to extract the granular clinical descriptors that guide treatment decisions remains inadequately characterized. Understanding the specific failure modes of LLM-based descriptor extraction is essential for the responsible development of clinical decision support systems. This study evaluated zero-shot LLM performance for clinical descriptor extraction in PE, characterizing misclassification patterns across multiple model architectures and developing a failure mode taxonomy to guide remediation strategies and clinical deployment decisions. Five zero-shot LLMs (gemma2:9b, qwen2.5:7b, mistral:7b, llama3.1:8b, llama3.2:3b) were evaluated for classification of four PE descriptors (anatomic location, temporal acuity, laterality, clot burden) across 1,463 CT radiology reports (33.6% PE prevalence) with physician-established gold standard labels from Hassan et al.’s (2025) dataset. Descriptor standardization mapped semantically equivalent terms to standard categories. Performance was assessed using confusion matrices, weighted Cohen's kappa, and Fleiss' kappa for cross-model agreement. Twenty-two engineered linguistic features were evaluated for associations with misclassification using Mann-Whitney U tests. Predictive models (LASSO, random forest, XGBoost) classified failure modes as complexity-driven (mean AUC ≥ 0.58) or concept-gap (mean AUC < 0.58). A prompt engineering experiment tested three prompt strategies (baseline, explicit category definitions, chain-of-thought reasoning) on gemma2:9b to empirically validate the failure mode taxonomy. A clear performance hierarchy emerged across descriptors: laterality achieved the highest mean accuracy (90.0%, QWK = 0.838), followed by acuity (82.7%, QWK = 0.610), location (74.3%, QWK = 0.629), and clot burden (53.9%, QWK = 0.400). The most striking finding was a universal 'blind spot' for moderate clot burden, with all five LLMs demonstrating near-zero recall for this intermediate category despite acceptable performance for low and high severity. Two distinct failure modes were identified: complexity-driven errors in acuity classification (mean prediction AUC = 0.643) were associated with linguistic features including mixed temporal language and high PE mention counts, while concept-gap errors in location and clot burden classification (mean AUCs < 0.58, at the clinical futility threshold) were unpredictable from text characteristics. The prompt engineering experiment provided direct experimental validation: chain-of-thought prompting dramatically shifted predictions toward the moderate category (reaching 72.4% recall), but at the severe cost of degrading low category recall (98.7% to 69.9%) and high category recall (79.5% to 33.3%), with overall accuracy declining from 66.6% to 62.2%. In contrast, explicit definition prompting improved acute-on-chronic acuity recall from 11.1% to 54.5% while maintaining overall accuracy (88.9% to 88.5%), confirming that complexity-driven errors respond to prompt engineering while concept-gap errors do not. This study presents the first systematic multi-model evaluation demonstrating that zero-shot LLMs exhibit fundamental, architecture-independent limitations in clinical descriptor extraction that persist across model families. The failure mode taxonomy distinguishing complexity-driven from concept-gap errors was experimentally validated through targeted prompt engineering, confirming that concept-gap errors in clot burden classification represent fundamental model limitations not addressable through prompt optimization alone. These findings provide an actionable framework for targeted remediation and graduated clinical deployment, underscoring that binary detection accuracy does not generalize to granular descriptor extraction and that hybrid approaches combining LLM capabilities with task-specific engineering remain necessary for reliable clinical information extraction.

Degree

thesis:*
Name thesis:degree_name
Master of Science
Level thesis:degree_level
masters
Discipline thesis:degree_discipline
Medical Sciences
Grantor
Boston University
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Webb, Joseph
Advisors dc:contributor.advisor
  • Hassan, Syed Moin
  • Bragdon, Beth C.

Subjects

dc:subject × 8

Rights

dc:rights
Statement dc:rights
  • Attribution-NonCommercial 4.0 International
Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/2144/53799

Chain of custody

source
Harvested from
Boston University
Base URL
open.bu.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Webb, Joseph. Systematic analysis of large language model descriptor classification in pulmonary embolism. masters thesis, Boston University, 2026. https://hdl.handle.net/2144/53799