  Recent research on model interpretability in natural language processing
extensively uses feature scoring methods for identifying which parts of the
input are the most important for a model to make a prediction (i.e. explanation
or rationale). However, previous research has shown that there is no clear best
scoring method across various text classification tasks while practitioners
typically have to make several other ad-hoc choices regarding the length and
the type of the rationale (e.g. short or long, contiguous or not). Inspired by
this, we propose a simple yet effective and flexible method that allows
selecting optimally for each data instance: (1) a feature scoring method; (2)
the length; and (3) the type of the rationale. Our method is inspired by input
erasure approaches to interpretability which assume that the most faithful
rationale for a prediction should be the one with the highest difference
between the model's output distribution using the full text and the text after
removing the rationale as input respectively. Evaluation on four standard text
classification datasets shows that our proposed method provides more faithful,
comprehensive and highly sufficient explanations compared to using a fixed
feature scoring method, rationale length and type. More importantly, we
demonstrate that a practitioner is not required to make any ad-hoc choices in
order to extract faithful rationales using our approach.
