Back to search

Virginia Tech

M3EC: A Multimodal Multidocument Benchmark for Event Extraction and Coreference Resolution

Abstract

dc:description.abstract

Understanding how the same real-world event is represented across text, video, and audio remains a fundamental challenge in information extraction. While prior datasets have explored events within individual modalities or limited modality pairs, a setting that combines all three modalities remains largely unexplored. We introduce M3EC (MultiModal Multidocument Event and Coreference), a new benchmark for multimodal, multidocument event extraction and coreference resolution. It is built from tightly related news clusters that combine eventful videos, audio files, and news articles. We also introduce a semi-automated annotation pipeline, AutoEventAnnotator, to generate automated annotations at scale. Additionally, we construct a human-annotated test set for robust evaluation. To our knowledge, this is the first benchmark to support cluster-level multidocument event extraction and event coreference across text, video, and audio. Evaluation with strong multimodal LLMs shows that the majority of them fail to extract valid events in video and particularly in audio. We further show that models fine-tuned on the annotations generated by our automated pipeline achieve consistent improvements, including more than +30 F1 on video event extraction, +15 F1 on audio event extraction, and +40 CoNLL F1 on end-to-end coreference resolution, highlighting the usefulness of M3EC for training and evaluation in multimodal event understanding.

Degree

thesis:*
Name thesis:degree_name
Master of Science
Level thesis:degree_level
masters
Discipline thesis:degree_discipline
Computer Science & Applications
Department dc:contributor.department
Computer Science and#38; Applications
Grantor dc:publisher
Virginia Tech
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Hakim, Zaber Ibn Abdul
Chair dc:contributor.committeechair
  • Thomas, Christopher Lee
Committee members dc:contributor.committeemember
  • Yanardag Delul, Pinar
  • Wang, Xuan

Subjects

dc:subject × 3

Rights

dc:rights
Statement dc:rights
  • In Copyright
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Other
vt_gsexam:47221

Chain of custody

source
Harvested from
Virginia Tech
Base URL
vtechworks.lib.vt.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Hakim, Zaber Ibn Abdul. M3EC: A Multimodal Multidocument Benchmark for Event Extraction and Coreference Resolution. masters thesis, Virginia Tech, 2026. https://hdl.handle.net/10919/143625