  Recent research at the intersection of AI explainability and fairness has
focused on how explanations can improve human-plus-AI task performance as
assessed by fairness measures. We propose to characterize what constitutes an
explanation that is itself "fair" -- an explanation that does not adversely
impact specific populations. We formulate a novel evaluation method of "fair
explanations" using not just accuracy and label time, but also psychological
impact of explanations on different user groups across many metrics (mental
discomfort, stereotype activation, and perceived workload). We apply this
method in the context of content moderation of potential hate speech, and its
differential impact on Asian vs. non-Asian proxy moderators, across explanation
approaches (saliency map and counterfactual explanation). We find that saliency
maps generally perform better and show less evidence of disparate impact
(group) and individual unfairness than counterfactual explanations.
  Content warning: This paper contains examples of hate speech and racially
discriminatory language. The authors do not support such content. Please
consider your risk of discomfort carefully before continuing reading!
