  The research field of Legal Natural Language Processing (NLP) has been very
active recently, with Legal Judgment Prediction (LJP) becoming one of the most
extensively studied tasks. To date, most publicly released LJP datasets
originate from countries with civil law. In this work, we release, for the
first time, a challenging LJP dataset focused on class action cases in the US.
It is the first dataset in the common law system that focuses on the harder and
more realistic task involving the complaints as input instead of the often used
facts summary written by the court. Additionally, we study the difficulty of
the task by collecting expert human predictions, showing that even human
experts can only reach 53% accuracy on this dataset. Our Longformer model
clearly outperforms the human baseline (63%), despite only considering the
first 2,048 tokens. Furthermore, we perform a detailed error analysis and find
that the Longformer model is significantly better calibrated than the human
experts. Finally, we publicly release the dataset and the code used for the
experiments.
