Back to search

Boston University

Robust multi-modal imaging in the presence of incomplete modalities

Abstract

dc:description.abstract

Multi-Channel Imaging (MCI) presents unique challenges beyond traditional RGB vision tasks, as the number and type of channels can vary drastically across domains such as microscopy and remote sensing. Existing neural networks are typically fixed-channel, limiting their reusability in settings where channel configurations change. To address this, we introduced the first systematic benchmark for channel-adaptive models in microscopy imaging, providing a varied-channel single-cell dataset and a biologically relevant evaluation framework, showing that channel-adaptive approaches improve generalization and efficiency over fixed-channel baselines. We then introduced Diverse Channel ViT (DiChaViT) to tackle the issue of redundant feature learning in MCI-ViTs by encouraging diversity in learned representations through a novel channel sampling strategy, architecture-agnostic regularization, and initialization techniques, achieving consistent gains across satellite and microscopy datasets. We proposed Channel Aware MAE - multichannel Vision Transformer (ChA-MAEViT) to further advance MCI representation learning by enhancing cross-channel interactions in Masked Autoencoders through dynamic channel-patch masking, memory tokens, hybrid token fusion, and a channel-aware decoder, leading to substantial improvements over prior MCI-ViTs. Finally, we conducted the first systematic study jointly examining pretraining objectives (DINOv3, MAE, SimCLR), channel encoding strategies (Bag-of-Channels vs. Multi-Channel), and pretraining data regimes for self-supervised multichannel microscopy. This investigation uncovered a fundamental flexibility–richness tradeoff: discriminative objectives paired with Bag-of-Channels encoding yield strong zero-shot generalization, while generative objectives paired with Multi-Channel encoding dominate after fine-tuning. Together, these works establish a foundation for robust, channel-adaptive visual understanding. In future work, we aim to extend these advances to multimodal settings by developing unified architectures capable of adapting to diverse inputs, such as imaging, text, and spatiotemporal signals. Such a framework would facilitate cross-modal learning and establish a foundation for versatile models that capture shared structures while preserving modality-specific characteristics.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy
Level thesis:degree_level
doctoral
Discipline thesis:degree_discipline
Computer Science
Grantor
Boston University
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Pham, Chau
Advisor dc:contributor.advisor
  • Plummer, Bryan A

Subjects

dc:subject × 6

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/2144/53691

Chain of custody

source
Harvested from
Boston University
Base URL
open.bu.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Pham, Chau. Robust multi-modal imaging in the presence of incomplete modalities. doctoral thesis, Boston University, 2026. https://hdl.handle.net/2144/53691