  In order to build reliable and trustworthy NLP applications, models need to
be both fair across different demographics and explainable. Usually these two
objectives, fairness and explainability, are optimized and/or examined
independently of each other. Instead, we argue that forthcoming, trustworthy
NLP systems should consider both. In this work, we perform a first study to
understand how they influence each other: do fair(er) models rely on more
plausible rationales? and vice versa. To this end, we conduct experiments on
two English multi-class text classification datasets, BIOS and ECtHR, that
provide information on gender and nationality, respectively, as well as
human-annotated rationales. We fine-tune pre-trained language models with
several methods for (i) bias mitigation, which aims to improve fairness; (ii)
rationale extraction, which aims to produce plausible explanations. We find
that bias mitigation algorithms do not always lead to fairer models. Moreover,
we discover that empirical fairness and explainability are orthogonal.
