{"id":{"repo_id":"msu","oai_identifier":"oai:d.lib.msu.edu:etd_52731"},"canonical_url":"https://search.ndltd.org/etd/msu/oai:d.lib.msu.edu:etd_52731","repository":{"repo_id":"msu","name":"Michigan State University","base_url":"https://d.lib.msu.edu/oai"},"display":{"title":"Learning, Seeing, and Highlighting with Multimodal Models","abstract":"Thesis (Ph.D.)--Michigan State University. Computer Science - Doctor of Philosophy, 2025","abstract_html":"Thesis (Ph.D.)--Michigan State University. Computer Science - Doctor of Philosophy, 2025","abstract_has_math":false,"creators":["Wang, Lan"],"institution":null,"degree_name":null,"degree_level":null,"degree_discipline":null,"degree_department":null,"school":null,"contributors":["Boddeti, Vishnu","Liu, Xiaoming","Kong, Yu","Lim, Sernam"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2025,"date_issued":"2025","date_published":"2025","updated_at":"2026-11-18T18:59:26Z","subjects":["Computer science"],"languages":["eng"],"rights":["Attribution 4.0 International"],"rights_urls":[],"identifier_entries":[{"key":"dc:identifier","label":"Identifier","values":["etd:52731","local:Wang_grad.msu_0128D_21702"],"render_values":[{"text":"etd:52731","href":null,"code":true},{"text":"local:Wang_grad.msu_0128D_21702","href":null,"code":true}]}],"person_identifier_entries":[]},"links":{"outbound_url":"https://doi.org/doi:10.25335/r9jq-ne21","outbound_label":"DOI","outbound_source":"dc:identifier","record_url":"https://doi.org/doi:10.25335/r9jq-ne21","record_label":"Repository record","record_kind":"doi","record_source":"dc:identifier","download_url":null,"download_label":null,"download_source":null,"access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://d.lib.msu.edu/oai?verb=GetRecord&metadataPrefix=oai_dc&identifier=oai%3Ad.lib.msu.edu%3Aetd_52731","prefix":"oai_dc"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Boddeti, Vishnu","Liu, Xiaoming","Kong, Yu","Lim, Sernam"]},{"key":"dc:creator","label":"Author","values":["Wang, Lan"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2025"]},{"key":"dc:relation","label":"Dc Relation","values":["Electronic Theses & Dissertations"]},{"key":"dc:type","label":"Dc Type","values":["text","Electronic dissertations"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Computer science"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Attribution 4.0 International"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["etd:52731","local:Wang_grad.msu_0128D_21702","https://doi.org/doi:10.25335/r9jq-ne21"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Thesis (Ph.D.)--Michigan State University. Computer Science - Doctor of Philosophy, 2025","Multimodal models integrate heterogeneous data sources such as text, images, audio, and video to support tasks that rely on information from multiple modalities. By jointly analyzing these modalities, they achieve more comprehensive and robust performance compared to unimodal approaches. Despite recent progress, several fundamental challenges in multimodal models remain unresolved. First, it is unclear how multimodal models understand and learn across different downstream tasks, and whether their learned representations are fair or can be less biased. Secondly, what does the ‘world’ imagined by multimodal models look like through generation, and can this synthesized information enhance their multimodal understanding ability? Finally, with the explosive growth of visual content, can multimodal models efficiently highlight useful information from massive data streams?In this thesis, We will investigate how multimodal models learn across different downstream tasks and achieve unbiased representations, see the imagined content before understanding, and highlight important content from extensive vision data.We first explore the learning of existing video language models in localization-related downstream tasks. Existing video temporal grounding (VTG) methods rely on pretrained features from trimmed videos, lacking temporal context. We propose a novel pretraining approach, which conducts an untrimmed pretraining with a Video-Text Similarity-based Grounding Module to enhance robustness against noise. In the second part of the thesis, We explore the fair representation learning in current vision-language models. Large pre-trained vision-language models such as CLIP provide compact and general-purpose representations of text and images. However, due to their pretraining, these models are at risk of both perpetuating and intensifying existing biases, as well as relying on spurious attributes. We propose a general approach by formulating the problem of jointly debiasing CLIP's image and text representations in reproducing kernel Hilbert spaces (RKHSs), which makes zero-shot predictions of CLIP more fair and robust to spurious correlations. In the third and fourth parts of the thesis, We explored the imagination ability of multimodal models. First, We investigates how multimodal models can generate videos. While existing video generation / editing tasks are limited to changes in objects, backgrounds, and styles, the new task aims to predict the necessary modifications to a given video to generate an altered video in response to a question (e.g., ``what if the man was running instead of walking?''). To solve this problem, We propose a comprehensive approach that first understands and reasons about the video content and questions, and then generates the corresponding imaginary video. Beyond imagination, We further investigate whether multimodal models can understanding better after seeing the imagined content. Specifically, existing multimodal visual retrieval tasks typically rely on a combination of visual and textual information. In contrast, We explored a different approach, which first makes the model imagine a pseudo target image based on the visual and textual inputs, and then using this imagined image to guide the multimodal retrieval process. Through seeing the imagined image, the model can perform retrieval more effectively.In the final part, We focus on leveraging multimodal models for long-video understanding tasks. As the scale of video data rapidly increases, the length of videos is no longer confined to just a few minutes but often extends to hours, posing new challenges incomputation and memory. To address this, We propose a semantic learning framework that enables models to identify and highlight contextually rich tokens, which capture the semantic flow and structure of the video over long time spans. This approach facilitates more effective downstream applications such as temporal grounding and video question answering.","Description based on online resource. Title from PDF t.p. (Michigan State University Fedora Repository, viewed ).","Includes bibliographical references."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf","1 online resource (148 p.) : ill."]},{"key":"dc:title","label":"Title","values":["Learning, Seeing, and Highlighting with Multimodal Models"]}]}],"canonical_facts":{"dc:contributor":["Boddeti, Vishnu","Liu, Xiaoming","Kong, Yu","Lim, Sernam"],"dc:creator":["Wang, Lan"],"dc:date":["2025"],"dc:description":["Thesis (Ph.D.)--Michigan State University. Computer Science - Doctor of Philosophy, 2025","Multimodal models integrate heterogeneous data sources such as text, images, audio, and video to support tasks that rely on information from multiple modalities. By jointly analyzing these modalities, they achieve more comprehensive and robust performance compared to unimodal approaches. Despite recent progress, several fundamental challenges in multimodal models remain unresolved. First, it is unclear how multimodal models understand and learn across different downstream tasks, and whether their learned representations are fair or can be less biased. Secondly, what does the ‘world’ imagined by multimodal models look like through generation, and can this synthesized information enhance their multimodal understanding ability? Finally, with the explosive growth of visual content, can multimodal models efficiently highlight useful information from massive data streams?In this thesis, We will investigate how multimodal models learn across different downstream tasks and achieve unbiased representations, see the imagined content before understanding, and highlight important content from extensive vision data.We first explore the learning of existing video language models in localization-related downstream tasks. Existing video temporal grounding (VTG) methods rely on pretrained features from trimmed videos, lacking temporal context. We propose a novel pretraining approach, which conducts an untrimmed pretraining with a Video-Text Similarity-based Grounding Module to enhance robustness against noise. In the second part of the thesis, We explore the fair representation learning in current vision-language models. Large pre-trained vision-language models such as CLIP provide compact and general-purpose representations of text and images. However, due to their pretraining, these models are at risk of both perpetuating and intensifying existing biases, as well as relying on spurious attributes. We propose a general approach by formulating the problem of jointly debiasing CLIP's image and text representations in reproducing kernel Hilbert spaces (RKHSs), which makes zero-shot predictions of CLIP more fair and robust to spurious correlations. In the third and fourth parts of the thesis, We explored the imagination ability of multimodal models. First, We investigates how multimodal models can generate videos. While existing video generation / editing tasks are limited to changes in objects, backgrounds, and styles, the new task aims to predict the necessary modifications to a given video to generate an altered video in response to a question (e.g., ``what if the man was running instead of walking?''). To solve this problem, We propose a comprehensive approach that first understands and reasons about the video content and questions, and then generates the corresponding imaginary video. Beyond imagination, We further investigate whether multimodal models can understanding better after seeing the imagined content. Specifically, existing multimodal visual retrieval tasks typically rely on a combination of visual and textual information. In contrast, We explored a different approach, which first makes the model imagine a pseudo target image based on the visual and textual inputs, and then using this imagined image to guide the multimodal retrieval process. Through seeing the imagined image, the model can perform retrieval more effectively.In the final part, We focus on leveraging multimodal models for long-video understanding tasks. As the scale of video data rapidly increases, the length of videos is no longer confined to just a few minutes but often extends to hours, posing new challenges incomputation and memory. To address this, We propose a semantic learning framework that enables models to identify and highlight contextually rich tokens, which capture the semantic flow and structure of the video over long time spans. This approach facilitates more effective downstream applications such as temporal grounding and video question answering.","Description based on online resource. Title from PDF t.p. (Michigan State University Fedora Repository, viewed ).","Includes bibliographical references."],"dc:format":["application/pdf","1 online resource (148 p.) : ill."],"dc:identifier":["etd:52731","local:Wang_grad.msu_0128D_21702","https://doi.org/doi:10.25335/r9jq-ne21"],"dc:language":["eng"],"dc:relation":["Electronic Theses & Dissertations"],"dc:rights":["Attribution 4.0 International"],"dc:subject":["Computer science"],"dc:title":["Learning, Seeing, and Highlighting with Multimodal Models"],"dc:type":["text","Electronic dissertations"]},"updated_at":"2026-11-18T18:59:26Z"}