  Pre-trained language models (PLMs) have shown impressive performance in
various language tasks. However, they are prone to spurious correlations, and
often generate illusory information. In real-world applications, PLMs should
justify decisions with formalized, coherent reasoning chains, but this
challenge remains under-explored. Cognitive psychology theorizes that humans
are capable of utilizing fast and intuitive heuristic thinking to make
decisions based on past experience, then rationalizing the decisions through
slower and deliberative analytic reasoning. We incorporate these interlinked
dual processes in fine-tuning and in-context learning with PLMs, applying them
to two language understanding tasks that require coherent physical commonsense
reasoning. We show that our proposed Heuristic-Analytic Reasoning (HAR)
strategies drastically improve the coherence of rationalizations for model
decisions, yielding state-of-the-art results on Tiered Reasoning for Intuitive
Physics (TRIP). We also find that this improved coherence is a direct result of
more faithful attention to relevant language context in each step of reasoning.
Our findings suggest that human-like reasoning strategies can effectively
improve the coherence and reliability of PLM reasoning.
