  The assessment of explainability in Legal Judgement Prediction (LJP) systems
is of paramount importance in building trustworthy and transparent systems,
particularly considering the reliance of these systems on factors that may lack
legal relevance or involve sensitive attributes. This study delves into the
realm of explainability and fairness in LJP models, utilizing Swiss Judgement
Prediction (SJP), the only available multilingual LJP dataset. We curate a
comprehensive collection of rationales that `support' and `oppose' judgement
from legal experts for 108 cases in German, French, and Italian. By employing
an occlusion-based explainability approach, we evaluate the explainability
performance of state-of-the-art monolingual and multilingual BERT-based LJP
models, as well as models developed with techniques such as data augmentation
and cross-lingual transfer, which demonstrated prediction performance
improvement. Notably, our findings reveal that improved prediction performance
does not necessarily correspond to enhanced explainability performance,
underscoring the significance of evaluating models from an explainability
perspective. Additionally, we introduce a novel evaluation framework, Lower
Court Insertion (LCI), which allows us to quantify the influence of lower court
information on model predictions, exposing current models' biases.
