Back to search

University of Glasgow

Adapting multimodal foundation models for recommendation

Abstract

dc:description.abstract

Multimodal foundation models (MFMs) now show strong representation and generation capabilities. Adapting them to recommender systems is a natural and increasingly central research direction, since items often contain rich multimodal signals such as text, images, videos, and audio. Yet MFMs are designed for general representation and generation tasks, making specific recommendation-oriented adaptation a key challenge. This thesis addresses this challenge from two connected perspectives: representation adaptation and generation adaptation. The first perspective studies end-to-end adaptation of continuous and discrete representations. Continuous representation adaptation jointly trains the MFM representation-learning pipeline and recommender through real-valued representations, making efficiency a central challenge. IISAN uses cached decoupled parameter-efficient fine-tuning to reduce computation overhead. Building on IISAN, we investigate adaptation depth and breadth. IISAN-Versa extends it to deeper, larger, asymmetric multimodal encoders, particularly pairing a large LLM-based text encoder with a smaller vision encoder. CROSSAN broadens it to multiple modality-specific foundation models through cross modal gated side adaptation. We then consider unified single-tower embedding models, whose general-domain capabilities pose a distinct challenge: existing multi-tower methods often become shallow side adapters and struggle with deep-layer adaptation. Stresa addresses this limitation through stream-aware residual adaptation. For discrete representation adaptation, jointly learning semantic-ID construction and the generative recommender introduces challenges, particularly non-differentiability and codebook collapse. DIGER tackles these using differentiable semantic IDs and an exploration mechanism that promotes codebook utilisation while maintaining end-to-end adaptation performance. The second perspective addresses a gap in the literature: recommendation oriented generation adaptation remains underexplored from platform and user perspectives. On the platform side, although generation can complete missing modalities for item presentation and recommendation model training, its effectiveness in recommendation settings lacks evaluation. MMPCBench addresses this gap by benchmarking MFMs on multimodal product content completion. On the user side, generated content can enter the recommendation ecosystem, yet little is known about optimising it for engagement. LLMPopcorn investigates the popularity of MFMgenerated content and explores LLMs for popularity oriented micro-video creation. Across seven works, this thesis studies MFM adaptation across representation and generation.

Degree

thesis:*
Level dc:type.qualificationlevel
PhD
Grantor dc:publisher.institution
University of Glasgow
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Fu, Junchen

Subjects

dc:subject × 1

Rights

Language dc:language
en

Identifiers

dc:identifier.*

Chain of custody

source
Harvested from
University of Glasgow
Base URL
theses.gla.ac.uk/cgi/oai2
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Fu, Junchen. Adapting multimodal foundation models for recommendation. PhD thesis, University of Glasgow, 2026. https://doi.org/10.5525/gla.thesis.86308