  Event extraction (EE) is a crucial task aiming at extracting events from
texts, which includes two subtasks: event detection (ED) and event argument
extraction (EAE). In this paper, we check the reliability of EE evaluations and
identify three major pitfalls: (1) The data preprocessing discrepancy makes the
evaluation results on the same dataset not directly comparable, but the data
preprocessing details are not widely noted and specified in papers. (2) The
output space discrepancy of different model paradigms makes different-paradigm
EE models lack grounds for comparison and also leads to unclear mapping issues
between predictions and annotations. (3) The absence of pipeline evaluation of
many EAE-only works makes them hard to be directly compared with EE works and
may not well reflect the model performance in real-world pipeline scenarios. We
demonstrate the significant influence of these pitfalls through comprehensive
meta-analyses of recent papers and empirical experiments. To avoid these
pitfalls, we suggest a series of remedies, including specifying data
preprocessing, standardizing outputs, and providing pipeline evaluation
results. To help implement these remedies, we develop a consistent evaluation
framework OMNIEVENT, which can be obtained from
https://github.com/THU-KEG/OmniEvent.
