  As increasingly complex AI systems are introduced into our daily lives, it
becomes important for such systems to be capable of explaining the rationale
for their decisions and allowing users to contest these decisions. A
significant hurdle to allowing for such explanatory dialogue could be the
vocabulary mismatch between the user and the AI system. This paper introduces
methods for providing contrastive explanations in terms of user-specified
concepts for sequential decision-making settings where the system's model of
the task may be best represented as an inscrutable model. We do this by
building partial symbolic models of a local approximation of the task that can
be leveraged to answer the user queries. We test these methods on a popular
Atari game (Montezuma's Revenge) and variants of Sokoban (a well-known planning
benchmark) and report the results of user studies to evaluate whether people
find explanations generated in this form useful.
