  Large Language Models (LLMs) have demonstrated proficiency in a wide array of
natural language processing tasks. However, its effectiveness over
discourse-level event relation extraction (ERE) tasks remains unexplored. In
this paper, we assess the effectiveness of LLMs in addressing discourse-level
ERE tasks characterized by lengthy documents and intricate relations
encompassing coreference, temporal, causal, and subevent types. Evaluation is
conducted using an commercial model, GPT-3.5, and an open-source model,
LLaMA-2. Our study reveals a notable underperformance of LLMs compared to the
baseline established through supervised learning. Although Supervised
Fine-Tuning (SFT) can improve LLMs performance, it does not scale well compared
to the smaller supervised baseline model. Our quantitative and qualitative
analysis shows that LLMs have several weaknesses when applied for extracting
event relations, including a tendency to fabricate event mentions, and failures
to capture transitivity rules among relations, detect long distance relations,
or comprehend contexts with dense event mentions.
