  The current fascination with large language models, or LLMs, derives from the
fact that many users lack the expertise to evaluate the quality of the
generated text. LLMs may therefore appear more capable than they actually are.
The dangerous combination of fluency and superficial plausibility leads to the
temptation to trust the generated text and creates the risk of overreliance.
Who would not trust perfect legalese? Relying recent findings in both technical
and legal scholarship, this Article counterbalances the overly optimistic
predictions as to the role of LLMs in legal practice. Integrating LLMs into
legal workstreams without a better comprehension of their limitations, will
create inefficiencies if not outright risks. Notwithstanding their
unprecedented ability to generate text, LLMs do not understand text. Without
the ability to understand meaning, LLMs will remain unable to use language, to
acquire knowledge and to perform complex reasoning tasks. Trained to model
language on the basis of stochastic word predictions, LLMs cannot distinguish
fact from fiction. Their knowledge of the law is limited to word strings
memorized in their parameters. It is also incomplete and largely incorrect.
LLMs operate at the level of word distributions, not at the level of verified
facts. The resulting propensity to hallucinate, to produce statements that are
incorrect but appear helpful and relevant, is alarming in high-risk areas like
legal services. At present, lawyers should beware of relying on text generated
by LLMs.
