  Realizing the recent advances in Natural Language Processing (NLP) to the
legal sector poses challenging problems such as extremely long sequence
lengths, specialized vocabulary that is usually only understood by legal
professionals, and high amounts of data imbalance. The recent surge of Large
Language Models (LLMs) has begun to provide new opportunities to apply NLP in
the legal domain due to their ability to handle lengthy, complex sequences.
Moreover, the emergence of domain-specific LLMs has displayed extremely
promising results on various tasks. In this study, we aim to quantify how
general LLMs perform in comparison to legal-domain models (be it an LLM or
otherwise). Specifically, we compare the zero-shot performance of three
general-purpose LLMs (ChatGPT-20b, LLaMA-2-70b, and Falcon-180b) on the LEDGAR
subset of the LexGLUE benchmark for contract provision classification. Although
the LLMs were not explicitly trained on legal data, we observe that they are
still able to classify the theme correctly in most cases. However, we find that
their mic-F1/mac-F1 performance is up to 19.2/26.8\% lesser than smaller models
fine-tuned on the legal domain, thus underscoring the need for more powerful
legal-domain LLMs.
