  Explainable AI (XAI) algorithms aim to help users understand how a machine
learning model makes predictions. To this end, many approaches explain which
input features are most predictive of a target label. However, such
explanations can still be puzzling to users (e.g., in product reviews, the word
"problems" is predictive of positive sentiment). If left unexplained, puzzling
explanations can have negative impacts. Explaining unintuitive associations
between an input feature and a target label is an underexplored area in XAI
research. We take an initial effort in this direction using unintuitive
associations learned by sentiment classifiers as a case study. We propose
approaches for (1) automatically detecting associations that can appear
unintuitive to users and (2) generating explanations to help users understand
why an unintuitive feature is predictive. Results from a crowdsourced study
(N=300) found that our proposed approaches can effectively detect and explain
predictive but unintuitive features in sentiment classification.
