  Event correlation reasoning infers whether a natural language paragraph
containing multiple events conforms to human common sense. For example, "Andrew
was very drowsy, so he took a long nap, and now he is very alert" is sound and
reasonable. In contrast, "Andrew was very drowsy, so he stayed up a long time,
now he is very alert" does not comply with human common sense. Such reasoning
capability is essential for many downstream tasks, such as script reasoning,
abductive reasoning, narrative incoherence, story cloze test, etc. However,
conducting event correlation reasoning is challenging due to a lack of large
amounts of diverse event-based knowledge and difficulty in capturing
correlation among multiple events. In this paper, we propose EventBERT, a
pre-trained model to encapsulate eventuality knowledge from unlabeled text.
Specifically, we collect a large volume of training examples by identifying
natural language paragraphs that describe multiple correlated events and
further extracting event spans in an unsupervised manner. We then propose three
novel event- and correlation-based learning objectives to pre-train an event
correlation model on our created training corpus. Empirical results show
EventBERT outperforms strong baselines on four downstream tasks, and achieves
SoTA results on most of them. Besides, it outperforms existing pre-trained
models by a large margin, e.g., 6.5~23%, in zero-shot learning of these tasks.
