  Current causal text mining datasets vary in objectives, data coverage, and
annotation schemes. These inconsistent efforts prevent modeling capabilities
and fair comparisons of model performance. Furthermore, few datasets include
cause-effect span annotations, which are needed for end-to-end causal relation
extraction. To address these issues, we propose UniCausal, a unified benchmark
for causal text mining across three tasks: (I) Causal Sequence Classification,
(II) Cause-Effect Span Detection and (III) Causal Pair Classification. We
consolidated and aligned annotations of six high quality, mainly
human-annotated, corpora, resulting in a total of 58,720, 12,144 and 69,165
examples for each task respectively. Since the definition of causality can be
subjective, our framework was designed to allow researchers to work on some or
all datasets and tasks. To create an initial benchmark, we fine-tuned BERT
pre-trained language models to each task, achieving 70.10% Binary F1, 52.42%
Macro F1, and 84.68% Binary F1 scores respectively.
