  Human-annotated labels and explanations are critical for training explainable
NLP models. However, unlike human-annotated labels whose quality is easier to
calibrate (e.g., with a majority vote), human-crafted free-form explanations
can be quite subjective. Before blindly using them as ground truth to train ML
models, a vital question needs to be asked: How do we evaluate a
human-annotated explanation's quality? In this paper, we build on the view that
the quality of a human-annotated explanation can be measured based on its
helpfulness (or impairment) to the ML models' performance for the desired NLP
tasks for which the annotations were collected. In comparison to the commonly
used Simulatability score, we define a new metric that can take into
consideration the helpfulness of an explanation for model performance at both
fine-tuning and inference. With the help of a unified dataset format, we
evaluated the proposed metric on five datasets (e.g., e-SNLI) against two model
architectures (T5 and BART), and the results show that our proposed metric can
objectively evaluate the quality of human-annotated explanations, while
Simulatability falls short.
