Aristotle University Of Thessaloniki (AUTH)
Multimodal deep learning methods for image-text verification in misinformation detection
Abstract
dc:descriptionThis doctoral dissertation investigates automated image-text verification for detecting multimodal misinformation, contributing to the development of novel deep learning methods, the creation of dedicated datasets, and evaluation under realistic conditions. Despite progress in the field, significant challenges remain, such as unimodal biases, reliance on superficial shortcut patterns, the retrieval of credible and relevant external evidence, and evidence leakage from published fact-checking articles. Part I focuses on internal image-text verification without external evidence. First, synthetic data generation strategies are compared and model generalization is evaluated on real-world misinformation. The analysis highlights unimodal biases in widely used benchmarks, leading to the creation of VERITE, a dataset that requires joint examination of image and text while mitigating reliance on shortcut signals. Furthermore, the Miscaption This! framework is proposed, leveraging Vision-Language Models to generate more realistic synthetic misinformation examples, and the LAMAR architecture is developed, utilizing multimodal latent reconstruction as an auxiliary verification signal. Part II explores evidence-based verification. To jointly estimate claim veracity and evidence relevance, RED-DOT is introduced—a Transformer-based architecture trained end-to-end for both tasks. Simultaneously, evidence-enabled shortcut learning is investigated, revealing cases where similarity to news articles enables high performance without genuine verification. In this context, MUSE, a multimodal similarity-based baseline, proves competitive with more complex verification approaches, uncovering limitations in existing benchmarks. Additionally, source credibility and leaked evidence from fact-checking articles are examined, both of which introduce unrealistic evaluation conditions. Part III expands research to real-world settings through the creation of X-POSE, a dataset of real posts from X (formerly Twitter) labeled via community consensus using Community Notes. Subsequently, TRENT (Evidence Triangulation) is proposed—an architecture that explicitly models relationships between image, text, and external evidence through dedicated cross-attention streams. Finally, the dissertation extends the study to image-text pairs involving AI-generated images through intent-aware classification and the S-HArM dataset, distinguishing misinformation from humor, satire, and artistic expression. The datasets developed in this dissertation and the corresponding implementation code are publicly available to facilitate further research and evaluation by the scientific community.
Degree
thesis:*- Grantor dc:publisher
- Aristotle University Of Thessaloniki (AUTH)
- Year dc:date
- 2026
Author and committee
dc:creator, dc:contributor.*- Authors dc:creator
-
- Papadopoulos, Stefanos-Iordanis
- Παπαδόπουλος, Στέφανος-Ιορδάνης
Subjects
dc:subject × 16- Βαθιά μάθηση
- Πολυτροπική μάθηση
- Ανίχνευση παραπληροφόρησης
- Επαλήθευση εικόνας-κειμένου
- Deep learning
- Multimodal learning
- Misinformation detection
- Image-text verification
- Επιστήμη Ηλεκτρονικών Υπολογιστών και Πληροφορική
- Φυσικές Επιστήμες
- Τεχνητή νοημοσύνη
- Διεπιστημονικές εφαρμογές επιστήμης ηλεκτρονικών υπολογιστών
- Computer and Information Sciences
- Natural Sciences
- Artificial Intelligence
- Computer Science Interdisciplinary Applications
Rights
- Language dc:language
- eng
Identifiers
dc:identifier.*- Identifier
- 10.12681/eadd/62874