  The introduction of large public legal datasets has brought about a
renaissance in legal NLP. Many of these datasets are comprised of legal
judgements - the product of judges deciding cases. This fact, together with the
way machine learning works, means that several legal NLP models are models of
judges. While some have argued for the automation of judges, in this position
piece, we argue that automating the role of the judge raises difficult ethical
challenges, in particular for common law legal systems. Our argument follows
from the social role of the judge in actively shaping the law, rather than
merely applying it. Since current NLP models come nowhere close to having the
facilities necessary for this task, they should not be used to automate judges.
Furthermore, even in the case the models could achieve human-level
capabilities, there would still be remaining ethical concerns inherent in the
automation of the legal process.
