  Free-text rationales play a pivotal role in explainable NLP, bridging the
knowledge and reasoning gaps behind a model's decision-making. However, due to
the diversity of potential reasoning paths and a corresponding lack of
definitive ground truth, their evaluation remains a challenge. Existing
evaluation metrics rely on the degree to which a rationale supports a target
label, but we find these fall short in evaluating rationales that inadvertently
leak the labels. To address this problem, we propose RORA, a Robust free-text
Rationale evaluation against label leakage. RORA quantifies the new information
supplied by a rationale to justify the label. This is achieved by assessing the
conditional V-information \citep{hewitt-etal-2021-conditional} with a
predictive family robust against leaky features that can be exploited by a
small model. RORA consistently outperforms existing approaches in evaluating
human-written, synthetic, or model-generated rationales, particularly
demonstrating robustness against label leakage. We also show that RORA aligns
well with human judgment, providing a more reliable and accurate measurement
across diverse free-text rationales.
