  Previous works on the fairness of toxic language classifiers compare the
output of models with different identity terms as input features but do not
consider the impact of other important concepts present in the context. Here,
besides identity terms, we take into account high-level latent features learned
by the classifier and investigate the interaction between these features and
identity terms. For a multi-class toxic language classifier, we leverage a
concept-based explanation framework to calculate the sensitivity of the model
to the concept of sentiment, which has been used before as a salient feature
for toxic language detection. Our results show that although for some classes,
the classifier has learned the sentiment information as expected, this
information is outweighed by the influence of identity terms as input features.
This work is a step towards evaluating procedural fairness, where unfair
processes lead to unfair outcomes. The produced knowledge can guide debiasing
techniques to ensure that important concepts besides identity terms are
well-represented in training datasets.
