Back to search

University of North Texas

An End-to-End Multi-Modal Deep Learning Framework for Transparent and Interpretable AI-assisted Neuroimaging

Abstract

dc:description

Brain tumours and intracranial haemorrhage are two time-sensitive neurological emergencies in which delays in diagnosis adversely impact patient care. Current artificial intelligence solutions evaluate one disease given one imaging modality and output classification scores with no accompanying structured rationale, anatomical grounding, or clinical report language. A single framework to jointly evaluate these diseases across multiple imaging modalities with interpretable features and automatically generated reports that clinicians can directly trace back through has not been established. This work further aims to be deployable in resource constrained clinical settings lacking subspecialty trained neuroimaging professionals. NeuroGAT-LM enables explainable multimodal neuroimaging deep learning in an end-to-end trainable model. Three streams encoding global, voxel, and spatial lesion knowl- edge (respectively: 3D ConvNet processing entire volume, 2D ConvNet measuring maximum intensity projection signatures, and graph neural network encoding lesion spatial relation- ships) are unified using equal-projection fusion to combine their embeddings, solving train loss degeneration towards stream dominance that occurred during development of an early prototype version (p stream -¿ 1; 62.4% ¿ 37.6%). Without knowledge from radiologist annotated labels, NeuroGAT-LM’s 19-headed concept reasoning module leverages automati- cally generated geometry-based pseudo-labels to generate modelled predictions of anatomical location, pathophysiological state, vascular territory, and cellular inference. Raw scan data entered as NIfTI format is transformed into structured clinical radiology report language using gradient-weighted class activation mapping (Grad-CAM) to spatially attribute concept presence to regions of the scan, before concept-level knowledge is deterministically normalized into human-readable form using hardcoded Look-Up Tables. Separate LLaVA-1.5 and Phi-2 language models are used to generate free-text and structured report output respectively, removing the possibility of hallucination in critical fields by divorcing deterministic concept classification from undeterministic natural language generation. The framework is trained incrementally in five stages, each driven by a numerically justified experimental result. Extensive experiments are performed on internal and external test sets covering both classes of disease and imaging modalities. This system shows that multi-task disease classification, well-formed concept reasoning, and rationale-grounded report generation are all achievable together using a single streamlined model that can run on one GPU in clinical environments around the world with limited resources.

Degree

thesis:*
Grantor dc:publisher
University of North Texas
Year dc:date
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Bhimavarapu, Sai Rushitha 2003-
Contributors dc:contributor
  • Bhowmick, Sanjukta
  • Tariq, Zeenat
  • Shah, Sayed K

Subjects

dc:subject × 3

Rights

dc:rights
Statement dc:rights
  • Public
  • Bhimavarapu, Sai Rushitha 2003-
  • Copyright
  • Copyright is held by the author, unless otherwise noted. All rights Reserved.
Language dc:language
English

Identifiers

dc:identifier.*
Identifier
local-cont-no: submission_4987
ark: ark:/67531/metadc2652492

Chain of custody

source
Harvested from
University of North Texas
Base URL
digital.library.unt.edu/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Bhimavarapu, Sai Rushitha 2003-. An End-to-End Multi-Modal Deep Learning Framework for Transparent and Interpretable AI-assisted Neuroimaging. University of North Texas, 2026. https://digital.library.unt.edu/ark:/67531/metadc2652492/