  The behavior of deep neural networks (DNNs) is hard to understand. This makes
it necessary to explore post hoc explanation methods. We conduct the first
comprehensive evaluation of explanation methods for NLP. To this end, we design
two novel evaluation paradigms that cover two important classes of NLP
problems: small context and large context problems. Both paradigms require no
manual annotation and are therefore broadly applicable. We also introduce
LIMSSE, an explanation method inspired by LIME that is designed for NLP. We
show empirically that LIMSSE, LRP and DeepLIFT are the most effective
explanation methods and recommend them for explaining DNNs in NLP.
