  Automatic legal text classification systems have been proposed in the
literature to address knowledge extraction from judgments and detect their
aspects. However, most of these systems are black boxes even when their models
are interpretable. This may raise concerns about their trustworthiness.
Accordingly, this work contributes with a system combining Natural Language
Processing (NLP) with Machine Learning (ML) to classify legal texts in an
explainable manner. We analyze the features involved in the decision and the
threshold bifurcation values of the decision paths of tree structures and
present this information to the users in natural language. This is the first
work on automatic analysis of legal texts combining NLP and ML along with
Explainable Artificial Intelligence techniques to automatically make the
models' decisions understandable to end users. Furthermore, legal experts have
validated our solution, and this knowledge has also been incorporated into the
explanation process as "expert-in-the-loop" dictionaries. Experimental results
on an annotated data set in law categories by jurisdiction demonstrate that our
system yields competitive classification performance, with accuracy values well
above 90%, and that its automatic explanations are easily understandable even
to non-expert users.
