  Modern recommender systems face an increasing need to explain their
recommendations. Despite considerable progress in this area, evaluating the
quality of explanations remains a significant challenge for researchers and
practitioners. Prior work mainly conducts human study to evaluate explanation
quality, which is usually expensive, time-consuming, and prone to human bias.
In this paper, we propose an offline evaluation method that can be computed
without human involvement. To evaluate an explanation, our method quantifies
its counterfactual impact on the recommendation. To validate the effectiveness
of our method, we carry out an online user study. We show that, compared to
conventional methods, our method can produce evaluation scores more correlated
with the real human judgments, and therefore can serve as a better proxy for
human evaluation. In addition, we show that explanations with high evaluation
scores are considered better by humans. Our findings highlight the promising
direction of using the counterfactual approach as one possible way to evaluate
recommendation explanations.
