Boston University
Systematic analysis of large language model descriptor classification in pulmonary embolism
Abstract
dc:description.abstractPulmonary embolism (PE) remains the third leading cause of cardiovascular mortality, affecting over 600,000 Americans annually. Large language models (LLMs) have demonstrated promising performance for binary PE detection from radiology reports, yet their capacity to extract the granular clinical descriptors that guide treatment decisions remains inadequately characterized. Understanding the specific failure modes of LLM-based descriptor extraction is essential for the responsible development of clinical decision support systems. This study evaluated zero-shot LLM performance for clinical descriptor extraction in PE, characterizing misclassification patterns across multiple model architectures and developing a failure mode taxonomy to guide remediation strategies and clinical deployment decisions. Five zero-shot LLMs (gemma2:9b, qwen2.5:7b, mistral:7b, llama3.1:8b, llama3.2:3b) were evaluated for classification of four PE descriptors (anatomic location, temporal acuity, laterality, clot burden) across 1,463 CT radiology reports (33.6% PE prevalence) with physician-established gold standard labels from Hassan et al.’s (2025) dataset. Descriptor standardization mapped semantically equivalent terms to standard categories. Performance was assessed using confusion matrices, weighted Cohen's kappa, and Fleiss' kappa for cross-model agreement. Twenty-two engineered linguistic features were evaluated for associations with misclassification using Mann-Whitney U tests. Predictive models (LASSO, random forest, XGBoost) classified failure modes as complexity-driven (mean AUC ≥ 0.58) or concept-gap (mean AUC < 0.58). A prompt engineering experiment tested three prompt strategies (baseline, explicit category definitions, chain-of-thought reasoning) on gemma2:9b to empirically validate the failure mode taxonomy. A clear performance hierarchy emerged across descriptors: laterality achieved the highest mean accuracy (90.0%, QWK = 0.838), followed by acuity (82.7%, QWK = 0.610), location (74.3%, QWK = 0.629), and clot burden (53.9%, QWK = 0.400). The most striking finding was a universal 'blind spot' for moderate clot burden, with all five LLMs demonstrating near-zero recall for this intermediate category despite acceptable performance for low and high severity. Two distinct failure modes were identified: complexity-driven errors in acuity classification (mean prediction AUC = 0.643) were associated with linguistic features including mixed temporal language and high PE mention counts, while concept-gap errors in location and clot burden classification (mean AUCs < 0.58, at the clinical futility threshold) were unpredictable from text characteristics. The prompt engineering experiment provided direct experimental validation: chain-of-thought prompting dramatically shifted predictions toward the moderate category (reaching 72.4% recall), but at the severe cost of degrading low category recall (98.7% to 69.9%) and high category recall (79.5% to 33.3%), with overall accuracy declining from 66.6% to 62.2%. In contrast, explicit definition prompting improved acute-on-chronic acuity recall from 11.1% to 54.5% while maintaining overall accuracy (88.9% to 88.5%), confirming that complexity-driven errors respond to prompt engineering while concept-gap errors do not. This study presents the first systematic multi-model evaluation demonstrating that zero-shot LLMs exhibit fundamental, architecture-independent limitations in clinical descriptor extraction that persist across model families. The failure mode taxonomy distinguishing complexity-driven from concept-gap errors was experimentally validated through targeted prompt engineering, confirming that concept-gap errors in clot burden classification represent fundamental model limitations not addressable through prompt optimization alone. These findings provide an actionable framework for targeted remediation and graduated clinical deployment, underscoring that binary detection accuracy does not generalize to granular descriptor extraction and that hybrid approaches combining LLM capabilities with task-specific engineering remain necessary for reliable clinical information extraction.
Degree
thesis:*- Name thesis:degree_name
- Master of Science
- Level thesis:degree_level
- masters
- Discipline thesis:degree_discipline
- Medical Sciences
- Grantor
- Boston University
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Webb, Joseph
- Advisors dc:contributor.advisor
-
- Hassan, Syed Moin
- Bragdon, Beth C.
Subjects
dc:subject × 8Rights
dc:rights- Statement dc:rights
-
- Attribution-NonCommercial 4.0 International
- Licence dc:rights.uri
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/2144/53799