Virginia Tech
M3EC: A Multimodal Multidocument Benchmark for Event Extraction and Coreference Resolution
Abstract
dc:description.abstractUnderstanding how the same real-world event is represented across text, video, and audio remains a fundamental challenge in information extraction. While prior datasets have explored events within individual modalities or limited modality pairs, a setting that combines all three modalities remains largely unexplored. We introduce M3EC (MultiModal Multidocument Event and Coreference), a new benchmark for multimodal, multidocument event extraction and coreference resolution. It is built from tightly related news clusters that combine eventful videos, audio files, and news articles. We also introduce a semi-automated annotation pipeline, AutoEventAnnotator, to generate automated annotations at scale. Additionally, we construct a human-annotated test set for robust evaluation. To our knowledge, this is the first benchmark to support cluster-level multidocument event extraction and event coreference across text, video, and audio. Evaluation with strong multimodal LLMs shows that the majority of them fail to extract valid events in video and particularly in audio. We further show that models fine-tuned on the annotations generated by our automated pipeline achieve consistent improvements, including more than +30 F1 on video event extraction, +15 F1 on audio event extraction, and +40 CoNLL F1 on end-to-end coreference resolution, highlighting the usefulness of M3EC for training and evaluation in multimodal event understanding.
Degree
thesis:*- Name thesis:degree_name
- Master of Science
- Level thesis:degree_level
- masters
- Discipline thesis:degree_discipline
- Computer Science & Applications
- Department dc:contributor.department
- Computer Science and#38; Applications
- Grantor dc:publisher
- Virginia Tech
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Hakim, Zaber Ibn Abdul
- Chair dc:contributor.committeechair
-
- Thomas, Christopher Lee
- Committee members dc:contributor.committeemember
-
- Yanardag Delul, Pinar
- Wang, Xuan
Subjects
dc:subject × 3Rights
dc:rights- Statement dc:rights
-
- In Copyright
- Licence dc:rights.uri
- Language dc:language.iso
- en
Identifiers
dc:identifier.*- Dc Identifier Other
- vt_gsexam:47221