  It is unclear if existing interpretations of deep neural network models
respond effectively to the needs of users. This paper summarizes the common
forms of explanations (such as feature attribution, decision rules, or probes)
used in over 200 recent papers about natural language processing (NLP), and
compares them against user questions collected in the XAI Question Bank. We
found that although users are interested in explanations for the road not taken
-- namely, why the model chose one result and not a well-defined, seemly
similar legitimate counterpart -- most model interpretations cannot answer
these questions.
