  Is AI set to redefine the legal profession? We argue that this claim is not
supported by the current evidence. We dive into AI's increasingly prevalent
roles in three types of legal tasks: information processing; tasks involving
creativity, reasoning, or judgment; and predictions about the future. We find
that the ease of evaluating legal applications varies greatly across legal
tasks, based on the ease of identifying correct answers and the observability
of information relevant to the task at hand. Tasks that would lead to the most
significant changes to the legal profession are also the ones most prone to
overoptimism about AI capabilities, as they are harder to evaluate. We make
recommendations for better evaluation and deployment of AI in legal contexts.
