Back to search

Aristotle University Of Thessaloniki (AUTH)

Multimodal deep learning methods for image-text verification in misinformation detection

Abstract

dc:description

This doctoral dissertation investigates automated image-text verification for detecting multimodal misinformation, contributing to the development of novel deep learning methods, the creation of dedicated datasets, and evaluation under realistic conditions. Despite progress in the field, significant challenges remain, such as unimodal biases, reliance on superficial shortcut patterns, the retrieval of credible and relevant external evidence, and evidence leakage from published fact-checking articles. Part I focuses on internal image-text verification without external evidence. First, synthetic data generation strategies are compared and model generalization is evaluated on real-world misinformation. The analysis highlights unimodal biases in widely used benchmarks, leading to the creation of VERITE, a dataset that requires joint examination of image and text while mitigating reliance on shortcut signals. Furthermore, the Miscaption This! framework is proposed, leveraging Vision-Language Models to generate more realistic synthetic misinformation examples, and the LAMAR architecture is developed, utilizing multimodal latent reconstruction as an auxiliary verification signal. Part II explores evidence-based verification. To jointly estimate claim veracity and evidence relevance, RED-DOT is introduced—a Transformer-based architecture trained end-to-end for both tasks. Simultaneously, evidence-enabled shortcut learning is investigated, revealing cases where similarity to news articles enables high performance without genuine verification. In this context, MUSE, a multimodal similarity-based baseline, proves competitive with more complex verification approaches, uncovering limitations in existing benchmarks. Additionally, source credibility and leaked evidence from fact-checking articles are examined, both of which introduce unrealistic evaluation conditions. Part III expands research to real-world settings through the creation of X-POSE, a dataset of real posts from X (formerly Twitter) labeled via community consensus using Community Notes. Subsequently, TRENT (Evidence Triangulation) is proposed—an architecture that explicitly models relationships between image, text, and external evidence through dedicated cross-attention streams. Finally, the dissertation extends the study to image-text pairs involving AI-generated images through intent-aware classification and the S-HArM dataset, distinguishing misinformation from humor, satire, and artistic expression. The datasets developed in this dissertation and the corresponding implementation code are publicly available to facilitate further research and evaluation by the scientific community.

Degree

thesis:*
Grantor dc:publisher
Aristotle University Of Thessaloniki (AUTH)
Year dc:date
2026

Author and committee

dc:creator, dc:contributor.*
Authors dc:creator
  • Papadopoulos, Stefanos-Iordanis
  • Παπαδόπουλος, Στέφανος-Ιορδάνης

Subjects

dc:subject × 16

Rights

Language dc:language
eng

Identifiers

dc:identifier.*
Identifier
10.12681/eadd/62874

Chain of custody

source
Harvested from
Greek National Archive of PhD Theses
Base URL
phdtheses.ekt.gr/eadd_oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Papadopoulos, Stefanos-Iordanis; Παπαδόπουλος, Στέφανος-Ιορδάνης. Multimodal deep learning methods for image-text verification in misinformation detection. Aristotle University Of Thessaloniki (AUTH), 2026. http://hdl.handle.net/10442/hedi/62874