  As deep neural models in NLP become more complex, and as a consequence
opaque, the necessity to interpret them becomes greater. A burgeoning interest
has emerged in rationalizing explanations to provide short and coherent
justifications for predictions. In this position paper, we advocate for a
formal framework for key concepts and properties about rationalizing
explanations to support their evaluation systematically. We also outline one
such formal framework, tailored to rationalizing explanations of increasingly
complex structures, from free-form explanations to deductive explanations, to
argumentative explanations (with the richest structure). Focusing on the
automated fact verification task, we provide illustrations of the use and
usefulness of our formalization for evaluating explanations, tailored to their
varying structures.
