  Large language models (LLMs) have demonstrated great potential for
domain-specific applications, such as the law domain. However, recent disputes
over GPT-4's law evaluation raise questions concerning their performance in
real-world legal tasks. To systematically investigate their competency in the
law, we design practical baseline solutions based on LLMs and test on the task
of legal judgment prediction. In our solutions, LLMs can work alone to answer
open questions or coordinate with an information retrieval (IR) system to learn
from similar cases or solve simplified multi-choice questions. We show that
similar cases and multi-choice options, namely label candidates, included in
prompts can help LLMs recall domain knowledge that is critical for expertise
legal reasoning. We additionally present an intriguing paradox wherein an IR
system surpasses the performance of LLM+IR due to limited gains acquired by
weaker LLMs from powerful IR systems. In such cases, the role of LLMs becomes
redundant. Our evaluation pipeline can be easily extended into other tasks to
facilitate evaluations in other domains. Code is available at
https://github.com/srhthu/LM-CompEval-Legal
