  Language models (LMs) have yielded impressive results on many language
reasoning tasks, but their unexpected errors raise doubts about their reasoning
abilities. In light of this, there is growing interest in finetuning/prompting
LMs with both task instances and their associated free-text rationales (FTRs),
which explain the correct reasoning process for predicting the correct task
output (i.e., how to be "right for the right reasons"). However, existing
finetuning methods fail to improve LM performance, while prompting needs
prohibitively large (i.e., >50B) LMs to work well. We propose KNIFE, which
shows that reasoning knowledge can be effectively distilled from FTRs into a
small (i.e., <1B) LM and improve the LM's performance. First, KNIFE finetunes a
teacher LM (given task input and FTR) to predict the task output, transferring
reasoning knowledge from the FTRs to the teacher's hidden states. Second, KNIFE
finetunes a student LM (given task input only) such that its hidden states are
aligned with the teacher's. Thus, the student is endowed with reasoning
knowledge but can be used for inference without direct FTR input. On two
question-answering datasets, KNIFE outperforms various finetuning and prompting
baselines in fully-supervised and low-resource settings. Also, we observe that
FTR quality is crucial to KNIFE's performance.
