  Neural language models (LMs) have achieved impressive results on various
language-based reasoning tasks by utilizing latent knowledge encoded in their
own pretrained parameters. To make this reasoning process more explicit, recent
works retrieve a rationalizing LM's internal knowledge by training or prompting
it to generate free-text rationales, which can be used to guide task
predictions made by either the same LM or a separate reasoning LM. However,
rationalizing LMs require expensive rationale annotation and/or computation,
without any assurance that their generated rationales improve LM task
performance or faithfully reflect LM decision-making. In this paper, we propose
PINTO, an LM pipeline that rationalizes via prompt-based learning, and learns
to faithfully reason over rationales via counterfactual regularization. First,
PINTO maps out a suitable reasoning process for the task input by prompting a
frozen rationalizing LM to generate a free-text rationale. Second, PINTO's
reasoning LM is fine-tuned to solve the task using the generated rationale as
context, while regularized to output less confident predictions when the
rationale is perturbed. Across four datasets, we show that PINTO significantly
improves the generalization ability of the reasoning LM, yielding higher
performance on both in-distribution and out-of-distribution test sets. Also, we
find that PINTO's rationales are more faithful to its task predictions than
those generated by competitive baselines.
