{"id":{"repo_id":"bu","oai_identifier":"oai:open.bu.edu:2144/49684"},"canonical_url":"https://search.ndltd.org/etd/bu/oai:open.bu.edu:2144/49684","repository":{"repo_id":"bu","name":"Boston University","base_url":"https://open.bu.edu/oai/request"},"display":{"title":"Towards effective and robust transformer-based models for biomedical applications","abstract":"Transformer-based deep neural networks have achieved great success in biomedical research and practice. Leveraging the strength of the self-attention mechanism and self-supervised learning schemes such as Masked Language Modeling (MLM), transformer-based Language Models (LMs) like Clinical BERT have shown impressive performance on clinical Natural Language Processing (NLP) tasks, while Vision Transformers (ViTs) also improve the traditional Convolutional Neural Network (CNN) models on clinical image analysis. These successes indicate the general sequence modeling power of transformers, which can be potentially extended to effectively analyze the biomedical sequential data in other modalities. On the other hand, ViTs trained under Empirical Risk Minimization (ERM) are known to be vulnerable to perturbations in the image, which limits their applications in biomedicine. This dissertation focuses on developing effective and robust transformer models and learning methods for a variety of biomedical applications. We propose a pre-training method to introduce prior knowledge from a medical knowledge base to transformer-based LMs, which can be implemented jointly with traditional MLM and improve the LM's ability on clinical text understanding. Inspired by the strength of MLM, we pre-train and fine-tune transformer-based protein LMs to achieve state-of-the-art MHC-peptide binding predictions. We also propose a causal Electronic Health Record (EHR) modeling scheme for general structured EHR modeling, which is implemented to train a Generative Pre-trained Transformer (GPT) and support real-world, unsupervised novel disease detection. To improve the robustness of transformers, we develop a distributionally robust deep learning framework, which significantly reduces the ViT classifier's error rate under image perturbations, and improves the stroke diagnosis accuracy under accelerated MRI settings. Our work not only practically improves the applicability and reliability of Artificial Intelligence (AI) in healthcare, but also provides general effective and robust sequence data processing frameworks for other research domains.","abstract_html":"Transformer-based deep neural networks have achieved great success in biomedical research and practice. Leveraging the strength of the self-attention mechanism and self-supervised learning schemes such as Masked Language Modeling (MLM), transformer-based Language Models (LMs) like Clinical BERT have shown impressive performance on clinical Natural Language Processing (NLP) tasks, while Vision Transformers (ViTs) also improve the traditional Convolutional Neural Network (CNN) models on clinical image analysis. These successes indicate the general sequence modeling power of transformers, which can be potentially extended to effectively analyze the biomedical sequential data in other modalities. On the other hand, ViTs trained under Empirical Risk Minimization (ERM) are known to be vulnerable to perturbations in the image, which limits their applications in biomedicine. This dissertation focuses on developing effective and robust transformer models and learning methods for a variety of biomedical applications. We propose a pre-training method to introduce prior knowledge from a medical knowledge base to transformer-based LMs, which can be implemented jointly with traditional MLM and improve the LM&#x27;s ability on clinical text understanding. Inspired by the strength of MLM, we pre-train and fine-tune transformer-based protein LMs to achieve state-of-the-art MHC-peptide binding predictions. We also propose a causal Electronic Health Record (EHR) modeling scheme for general structured EHR modeling, which is implemented to train a Generative Pre-trained Transformer (GPT) and support real-world, unsupervised novel disease detection. To improve the robustness of transformers, we develop a distributionally robust deep learning framework, which significantly reduces the ViT classifier&#x27;s error rate under image perturbations, and improves the stroke diagnosis accuracy under accelerated MRI settings. Our work not only practically improves the applicability and reliability of Artificial Intelligence (AI) in healthcare, but also provides general effective and robust sequence data processing frameworks for other research domains.","abstract_has_math":false,"creators":["Hao, Boran"],"institution":"Boston University","degree_name":"Doctor of Philosophy","degree_level":"doctoral","degree_discipline":"Electrical & Computer Engineering","degree_department":null,"school":null,"contributors":[],"advisors":["Paschalidis, Ioannis Ch."],"committee_chairs":[],"committee_members":[],"year":2025,"date_issued":"2025","date_published":"2025","updated_at":"2026-11-18T20:18:58Z","subjects":["Electrical engineering","Biomedical engineering","Computer science","Biomedicine","Computer vision","Deep learning","Natural language processing","Sequence modeling","Transformer"],"languages":["en_US"],"rights":["Attribution 4.0 International"],"rights_urls":["http://creativecommons.org/licenses/by/4.0/"],"identifier_entries":[],"person_identifier_entries":[{"key":"dc:identifier.orcid","label":"ORCID","values":["0000-0001-7922-0513"],"render_values":[{"text":"0000-0001-7922-0513","href":"https://orcid.org/0000-0001-7922-0513","code":true}]}]},"links":{"outbound_url":"https://hdl.handle.net/2144/49684","outbound_label":"Handle","outbound_source":"dc:identifier.uri","record_url":"https://hdl.handle.net/2144/49684","record_label":"Repository record","record_kind":"handle","record_source":"dc:identifier.uri","download_url":null,"download_label":null,"download_source":null,"access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://open.bu.edu/oai/request?verb=GetRecord&metadataPrefix=dim&identifier=oai%3Aopen.bu.edu%3A2144%2F49684","prefix":"dim"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor.advisor","label":"Advisor","values":["Paschalidis, Ioannis Ch."]},{"key":"dc:creator","label":"Author","values":["Hao, Boran"]},{"key":"dc:identifier.orcid","label":"ORCID","values":["0000-0001-7922-0513"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.accessioned","label":"Dc Date Accessioned","values":["2025-02-03T19:27:47Z"]},{"key":"dc:date.available","label":"Dc Date Available","values":["2025-02-03T19:27:47Z"]},{"key":"dc:date.issued","label":"Date","values":["2025"]},{"key":"dc:date.updated","label":"Dc Date Updated","values":["2025-01-29T02:03:29Z"]},{"key":"dc:type","label":"Dc Type","values":["Thesis/Dissertation"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Electrical & Computer Engineering"]},{"key":"thesis:degree_level","label":"Degree Level","values":["doctoral"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Doctor of Philosophy"]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["Boston University"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Electrical engineering","Biomedical engineering","Computer science","Biomedicine","Computer vision","Deep learning","Natural language processing","Sequence modeling","Transformer"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language.iso","label":"Language (ISO)","values":["en_US"]},{"key":"dc:rights","label":"Dc Rights","values":["Attribution 4.0 International"]},{"key":"dc:rights.uri","label":"Rights URI","values":["http://creativecommons.org/licenses/by/4.0/"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier.uri","label":"Identifier URI","values":["https://hdl.handle.net/2144/49684"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["2025"]},{"key":"dc:description.abstract","label":"Abstract","values":["Transformer-based deep neural networks have achieved great success in biomedical research and practice. Leveraging the strength of the self-attention mechanism and self-supervised learning schemes such as Masked Language Modeling (MLM), transformer-based Language Models (LMs) like Clinical BERT have shown impressive performance on clinical Natural Language Processing (NLP) tasks, while Vision Transformers (ViTs) also improve the traditional Convolutional Neural Network (CNN) models on clinical image analysis. These successes indicate the general sequence modeling power of transformers, which can be potentially extended to effectively analyze the biomedical sequential data in other modalities. On the other hand, ViTs trained under Empirical Risk Minimization (ERM) are known to be vulnerable to perturbations in the image, which limits their applications in biomedicine. This dissertation focuses on developing effective and robust transformer models and learning methods for a variety of biomedical applications. We propose a pre-training method to introduce prior knowledge from a medical knowledge base to transformer-based LMs, which can be implemented jointly with traditional MLM and improve the LM's ability on clinical text understanding. Inspired by the strength of MLM, we pre-train and fine-tune transformer-based protein LMs to achieve state-of-the-art MHC-peptide binding predictions. We also propose a causal Electronic Health Record (EHR) modeling scheme for general structured EHR modeling, which is implemented to train a Generative Pre-trained Transformer (GPT) and support real-world, unsupervised novel disease detection. To improve the robustness of transformers, we develop a distributionally robust deep learning framework, which significantly reduces the ViT classifier's error rate under image perturbations, and improves the stroke diagnosis accuracy under accelerated MRI settings. Our work not only practically improves the applicability and reliability of Artificial Intelligence (AI) in healthcare, but also provides general effective and robust sequence data processing frameworks for other research domains."]},{"key":"dc:title","label":"Title","values":["Towards effective and robust transformer-based models for biomedical applications"]}]}],"canonical_facts":{"dc:contributor.advisor":["Paschalidis, Ioannis Ch."],"dc:creator":["Hao, Boran"],"dc:date.accessioned":["2025-02-03T19:27:47Z"],"dc:date.available":["2025-02-03T19:27:47Z"],"dc:date.issued":["2025"],"dc:date.updated":["2025-01-29T02:03:29Z"],"dc:description":["2025"],"dc:description.abstract":["Transformer-based deep neural networks have achieved great success in biomedical research and practice. Leveraging the strength of the self-attention mechanism and self-supervised learning schemes such as Masked Language Modeling (MLM), transformer-based Language Models (LMs) like Clinical BERT have shown impressive performance on clinical Natural Language Processing (NLP) tasks, while Vision Transformers (ViTs) also improve the traditional Convolutional Neural Network (CNN) models on clinical image analysis. These successes indicate the general sequence modeling power of transformers, which can be potentially extended to effectively analyze the biomedical sequential data in other modalities. On the other hand, ViTs trained under Empirical Risk Minimization (ERM) are known to be vulnerable to perturbations in the image, which limits their applications in biomedicine. This dissertation focuses on developing effective and robust transformer models and learning methods for a variety of biomedical applications. We propose a pre-training method to introduce prior knowledge from a medical knowledge base to transformer-based LMs, which can be implemented jointly with traditional MLM and improve the LM's ability on clinical text understanding. Inspired by the strength of MLM, we pre-train and fine-tune transformer-based protein LMs to achieve state-of-the-art MHC-peptide binding predictions. We also propose a causal Electronic Health Record (EHR) modeling scheme for general structured EHR modeling, which is implemented to train a Generative Pre-trained Transformer (GPT) and support real-world, unsupervised novel disease detection. To improve the robustness of transformers, we develop a distributionally robust deep learning framework, which significantly reduces the ViT classifier's error rate under image perturbations, and improves the stroke diagnosis accuracy under accelerated MRI settings. Our work not only practically improves the applicability and reliability of Artificial Intelligence (AI) in healthcare, but also provides general effective and robust sequence data processing frameworks for other research domains."],"dc:identifier.orcid":["0000-0001-7922-0513"],"dc:identifier.uri":["https://hdl.handle.net/2144/49684"],"dc:language.iso":["en_US"],"dc:rights":["Attribution 4.0 International"],"dc:rights.uri":["http://creativecommons.org/licenses/by/4.0/"],"dc:subject":["Electrical engineering","Biomedical engineering","Computer science","Biomedicine","Computer vision","Deep learning","Natural language processing","Sequence modeling","Transformer"],"dc:title":["Towards effective and robust transformer-based models for biomedical applications"],"dc:type":["Thesis/Dissertation"],"thesis:degree_discipline":["Electrical & Computer Engineering"],"thesis:degree_level":["doctoral"],"thesis:degree_name":["Doctor of Philosophy"],"thesis:institution_name":["Boston University"]},"updated_at":"2026-11-18T20:18:58Z"}