Back to search

University of Waterloo

Visual Segmentation with Explanations for Medical Decision Making: A CNN Architecture with a Transformer-Attention Extension

Abstract

dc:description.abstract

Medical image segmentation is essential for assisting medical professionals in locating anomalies in images. However, the lack of explainability in current segmentation frameworks limits clinical trust and adoption. This thesis presents two complementary frameworks for explainable medical image segmentation, demonstrating that accuracy and interpretability can be achieved simultaneously through innovative architectural design. The first framework introduces an attention-based CNN architecture that generates model-specific visual explanations through spatial attention gates integrated directly into the network. The proposed model achieves a Dice score of 0.8621 on the Kvasir-SEG polyp segmentation dataset, outperforming all evaluated explainable models, while producing attention heatmaps that faithfully reflect the model's decision-making process. The second framework advances this by introducing a dual encoder architecture combining a pretrained ResNet-34 CNN encoder with a pretrained Swin Transformer encoder, fused through a learned directional attention-gated mechanism at multiple scales. The dual encoder achieves a Dice score of 0.907 on Kvasir-SEG, outperforming all single encoder baselines, while generating richer multi-scale visual explanations that reflect the complementary contributions of both encoders. Finally, this thesis outlines a pathway in detail towards a fully multimodal explainability system, integrating textual explanations through SigLIP, Retrieval Augmented Generation, and a Large Language Model alongside the visual heatmaps. We explore future directions regarding breast cancer and echocardiogram segmentation, more specifically applications for ejection fraction computation,and insights gained from trials and errors that led to the innovative designs of the thesis.

Degree

thesis:*
Grantor dc:publisher
University of Waterloo
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Khalaf, Mahmoud

Subjects

dc:subject × 5

Rights

Language dc:language.iso
en

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/10012/23338

Chain of custody

source
Harvested from
University of Waterloo
Base URL
dspacemainprd01.lib.uwaterloo.ca/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Khalaf, Mahmoud. Visual Segmentation with Explanations for Medical Decision Making: A CNN Architecture with a Transformer-Attention Extension. University of Waterloo, 2026. https://hdl.handle.net/10012/23338