  Events describe happenings in our world that are of importance. Naturally,
understanding events mentioned in multimedia content and how they are related
forms an important way of comprehending our world. Existing literature can
infer if events across textual and visual (video) domains are identical (via
grounding) and thus, on the same semantic level. However, grounding fails to
capture the intricate cross-event relations that exist due to the same events
being referred to on many semantic levels. For example, in Figure 1, the
abstract event of "war" manifests at a lower semantic level through subevents
"tanks firing" (in video) and airplane "shot" (in text), leading to a
hierarchical, multimodal relationship between the events.
  In this paper, we propose the task of extracting event hierarchies from
multimodal (video and text) data to capture how the same event manifests itself
in different modalities at different semantic levels. This reveals the
structure of events and is critical to understanding them. To support research
on this task, we introduce the Multimodal Hierarchical Events (MultiHiEve)
dataset. Unlike prior video-language datasets, MultiHiEve is composed of news
video-article pairs, which makes it rich in event hierarchies. We densely
annotate a part of the dataset to construct the test benchmark. We show the
limitations of state-of-the-art unimodal and multimodal baselines on this task.
Further, we address these limitations via a new weakly supervised model,
leveraging only unannotated video-article pairs from MultiHiEve. We perform a
thorough evaluation of our proposed method which demonstrates improved
performance on this task and highlight opportunities for future research.
