  Following how humans communicate, free-text rationales aim to use natural
language to explain neural language model (LM) behavior. However, free-text
rationales' unconstrained nature makes them prone to hallucination, so it is
important to have metrics for free-text rationale quality. Existing free-text
rationale metrics measure how consistent the rationale is with the LM's
predicted label, but there is no protocol for assessing such metrics'
reliability. Thus, we propose FRAME, a framework for evaluating rationale-label
consistency (RLC) metrics for free-text rationales. FRAME is based on three
axioms: (1) good metrics should yield highest scores for reference rationales,
which maximize RLC by construction; (2) good metrics should be appropriately
sensitive to semantic perturbation of rationales; and (3) good metrics should
be robust to variation in the LM's task performance. Across three text
classification datasets, we show that existing RLC metrics cannot satisfy all
three FRAME axioms, since they are implemented via model pretraining which
muddles the metric's signal. Then, we introduce a non-pretraining RLC metric
that greatly outperforms baselines on (1) and (3), while performing
competitively on (2). Finally, we discuss the limitations of using RLC to
evaluate free-text rationales.
