  Free-form rationales aim to aid model interpretability by supplying the
background knowledge that can help understand model decisions. Crowdsourced
rationales are provided for commonsense QA instances in popular datasets such
as CoS-E and ECQA, but their utility remains under-investigated. We present
human studies which show that ECQA rationales indeed provide additional
background information to understand a decision, while over 88% of CoS-E
rationales do not. Inspired by this finding, we ask: can the additional context
provided by free-form rationales benefit models, similar to human users? We
investigate the utility of rationales as an additional source of supervision,
by varying the quantity and quality of rationales during training. After
controlling for instances where rationales leak the correct answer while not
providing additional background knowledge, we find that incorporating only 5%
of rationales during training can boost model performance by 47.22% for CoS-E
and 57.14% for ECQA during inference. Moreover, we also show that rationale
quality matters: compared to crowdsourced rationales, T5-generated rationales
provide not only weaker supervision to models, but are also not helpful for
humans in aiding model interpretability.
