Back to search

Boston University

Towards effective and robust transformer-based models for biomedical applications

Abstract

dc:description.abstract

Transformer-based deep neural networks have achieved great success in biomedical research and practice. Leveraging the strength of the self-attention mechanism and self-supervised learning schemes such as Masked Language Modeling (MLM), transformer-based Language Models (LMs) like Clinical BERT have shown impressive performance on clinical Natural Language Processing (NLP) tasks, while Vision Transformers (ViTs) also improve the traditional Convolutional Neural Network (CNN) models on clinical image analysis. These successes indicate the general sequence modeling power of transformers, which can be potentially extended to effectively analyze the biomedical sequential data in other modalities. On the other hand, ViTs trained under Empirical Risk Minimization (ERM) are known to be vulnerable to perturbations in the image, which limits their applications in biomedicine. This dissertation focuses on developing effective and robust transformer models and learning methods for a variety of biomedical applications. We propose a pre-training method to introduce prior knowledge from a medical knowledge base to transformer-based LMs, which can be implemented jointly with traditional MLM and improve the LM's ability on clinical text understanding. Inspired by the strength of MLM, we pre-train and fine-tune transformer-based protein LMs to achieve state-of-the-art MHC-peptide binding predictions. We also propose a causal Electronic Health Record (EHR) modeling scheme for general structured EHR modeling, which is implemented to train a Generative Pre-trained Transformer (GPT) and support real-world, unsupervised novel disease detection. To improve the robustness of transformers, we develop a distributionally robust deep learning framework, which significantly reduces the ViT classifier's error rate under image perturbations, and improves the stroke diagnosis accuracy under accelerated MRI settings. Our work not only practically improves the applicability and reliability of Artificial Intelligence (AI) in healthcare, but also provides general effective and robust sequence data processing frameworks for other research domains.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy
Level thesis:degree_level
doctoral
Discipline thesis:degree_discipline
Electrical & Computer Engineering
Grantor
Boston University
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Hao, Boran
Advisor dc:contributor.advisor
  • Paschalidis, Ioannis Ch.
ORCID dc:identifier.orcid

Subjects

dc:subject × 9

Rights

dc:rights
Statement dc:rights
  • Attribution 4.0 International
Language dc:language.iso
en_US

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/2144/49684

Chain of custody

source
Harvested from
Boston University
Base URL
open.bu.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Hao, Boran. Towards effective and robust transformer-based models for biomedical applications. doctoral thesis, Boston University, 2025. https://hdl.handle.net/2144/49684