  NLP practitioners often want to take existing trained models and apply them
to data from new domains. While fine-tuning or few-shot learning can be used to
adapt a base model, there is no single recipe for making these techniques work;
moreover, one may not have access to the original model weights if it is
deployed as a black box. We study how to improve a black box model's
performance on a new domain by leveraging explanations of the model's behavior.
Our approach first extracts a set of features combining human intuition about
the task with model attributions generated by black box interpretation
techniques, then uses a simple calibrator, in the form of a classifier, to
predict whether the base model was correct or not. We experiment with our
method on two tasks, extractive question answering and natural language
inference, covering adaptation from several pairs of domains with limited
target-domain data. The experimental results across all the domain pairs show
that explanations are useful for calibrating these models, boosting accuracy
when predictions do not have to be returned on every example. We further show
that the calibration model transfers to some extent between tasks.
