  By explaining how humans would solve a given task, human rationales can
provide strong learning signal for neural language models (LMs). Explanation
regularization (ER) aims to improve LM generalization by pushing the LM's
machine rationales (Which input tokens did the LM focus on?) to align with
human rationales (Which input tokens would humans focus on?). Though prior
works primarily study ER via in-distribution (ID) evaluation,
out-of-distribution (OOD) generalization is often more critical in real-world
scenarios, yet ER's effect on OOD generalization has been underexplored. In
this paper, we introduce ER-Test, a framework for evaluating ER models' OOD
generalization along three dimensions: unseen dataset tests, contrast set
tests, and functional tests. Using ER-Test, we extensively analyze how ER
models' OOD generalization varies with different ER design choices. Across two
tasks and six datasets, ER-Test shows that ER has little impact on ID
performance but can yield large OOD performance gains. Also, we find that ER
can improve OOD performance even with limited rationale supervision. ER-Test's
results help demonstrate ER's utility and establish best practices for using ER
effectively.
