  Seeking legal advice is often expensive. Recent advancements in machine
learning for solving complex problems can be leveraged to help make legal
services more accessible to the public. However, real-life applications
encounter significant challenges. State-of-the-art language models are growing
increasingly large, making parameter-efficient learning increasingly important.
Unfortunately, parameter-efficient methods perform poorly with small amounts of
data, which are common in the legal domain (where data labelling costs are
high). To address these challenges, we propose parameter-efficient legal domain
adaptation, which uses vast unsupervised legal data from public legal forums to
perform legal pre-training. This method exceeds or matches the fewshot
performance of existing models such as LEGAL-BERT on various legal tasks while
tuning only approximately 0.1% of model parameters. Additionally, we show that
our method can achieve calibration comparable to existing methods across
several tasks. To the best of our knowledge, this work is among the first to
explore parameter-efficient methods of tuning language models in the legal
domain.
