Boston University
Robust multi-modal imaging in the presence of incomplete modalities
Abstract
dc:description.abstractMulti-Channel Imaging (MCI) presents unique challenges beyond traditional RGB vision tasks, as the number and type of channels can vary drastically across domains such as microscopy and remote sensing. Existing neural networks are typically fixed-channel, limiting their reusability in settings where channel configurations change. To address this, we introduced the first systematic benchmark for channel-adaptive models in microscopy imaging, providing a varied-channel single-cell dataset and a biologically relevant evaluation framework, showing that channel-adaptive approaches improve generalization and efficiency over fixed-channel baselines. We then introduced Diverse Channel ViT (DiChaViT) to tackle the issue of redundant feature learning in MCI-ViTs by encouraging diversity in learned representations through a novel channel sampling strategy, architecture-agnostic regularization, and initialization techniques, achieving consistent gains across satellite and microscopy datasets. We proposed Channel Aware MAE - multichannel Vision Transformer (ChA-MAEViT) to further advance MCI representation learning by enhancing cross-channel interactions in Masked Autoencoders through dynamic channel-patch masking, memory tokens, hybrid token fusion, and a channel-aware decoder, leading to substantial improvements over prior MCI-ViTs. Finally, we conducted the first systematic study jointly examining pretraining objectives (DINOv3, MAE, SimCLR), channel encoding strategies (Bag-of-Channels vs. Multi-Channel), and pretraining data regimes for self-supervised multichannel microscopy. This investigation uncovered a fundamental flexibility–richness tradeoff: discriminative objectives paired with Bag-of-Channels encoding yield strong zero-shot generalization, while generative objectives paired with Multi-Channel encoding dominate after fine-tuning. Together, these works establish a foundation for robust, channel-adaptive visual understanding. In future work, we aim to extend these advances to multimodal settings by developing unified architectures capable of adapting to diverse inputs, such as imaging, text, and spatiotemporal signals. Such a framework would facilitate cross-modal learning and establish a foundation for versatile models that capture shared structures while preserving modality-specific characteristics.
Degree
thesis:*- Name thesis:degree_name
- Doctor of Philosophy
- Level thesis:degree_level
- doctoral
- Discipline thesis:degree_discipline
- Computer Science
- Grantor
- Boston University
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Pham, Chau
- Advisor dc:contributor.advisor
-
- Plummer, Bryan A
Subjects
dc:subject × 6Rights
dc:rights- Statement dc:rights
-
- Attribution 4.0 International
- Licence dc:rights.uri
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/2144/53691