  Legal case retrieval aims to help legal workers find relevant cases related
to their cases at hand, which is important for the guarantee of fairness and
justice in legal judgments. While recent advances in neural retrieval methods
have significantly improved the performance of open-domain retrieval tasks
(e.g., Web search), their advantages have not been observed in legal case
retrieval due to their thirst for annotated data. As annotating large-scale
training data in legal domains is prohibitive due to the need for domain
expertise, traditional search techniques based on lexical matching such as
TF-IDF, BM25, and Query Likelihood are still prevalent in legal case retrieval
systems. While previous studies have designed several pre-training methods for
IR models in open-domain tasks, these methods are usually suboptimal in legal
case retrieval because they cannot understand and capture the key knowledge and
data structures in the legal corpus. To this end, we propose a novel
pre-training framework named Caseformer that enables the pre-trained models to
learn legal knowledge and domain-specific relevance information in legal case
retrieval without any human-labeled data. Through three unsupervised learning
tasks, Caseformer is able to capture the special language, document structure,
and relevance patterns of legal case documents, making it a strong backbone for
downstream legal case retrieval tasks. Experimental results show that our model
has achieved state-of-the-art performance in both zero-shot and full-data
fine-tuning settings. Also, experiments on both Chinese and English legal
datasets demonstrate that the effectiveness of Caseformer is
language-independent in legal case retrieval.
