  There has been mounting evidence that pretrained language models fine-tuned
on large and diverse supervised datasets can transfer well to a variety of
out-of-domain tasks. In this work, we investigate this transfer ability to the
legal domain. For that, we participated in the legal case entailment task of
COLIEE 2021, in which we use such models with no adaptations to the target
domain. Our submissions achieved the highest scores, surpassing the second-best
team by more than six percentage points. Our experiments confirm a
counter-intuitive result in the new paradigm of pretrained language models:
given limited labeled data, models with little or no adaptation to the target
task can be more robust to changes in the data distribution than models
fine-tuned on it. Code is available at https://github.com/neuralmind-ai/coliee.
