  Multi-step reasoning ability is fundamental to many natural language tasks,
yet it is unclear what constitutes a good reasoning chain and how to evaluate
them. Most existing methods focus solely on whether the reasoning chain leads
to the correct conclusion, but this answer-oriented view may confound reasoning
quality with other spurious shortcuts to predict the answer. To bridge this
gap, we evaluate reasoning chains by viewing them as informal proofs that
derive the final answer. Specifically, we propose ReCEval (Reasoning Chain
Evaluation), a framework that evaluates reasoning chains via two key
properties: (1) correctness, i.e., each step makes a valid inference based on
information contained within the step, preceding steps, and input context, and
(2) informativeness, i.e., each step provides new information that is helpful
towards deriving the generated answer. We evaluate these properties by
developing metrics using natural language inference models and V-Information.
On multiple datasets, we show that ReCEval effectively identifies various error
types and yields notable improvements compared to prior methods. We analyze the
impact of step boundaries, and previous steps on evaluating correctness and
demonstrate that our informativeness metric captures the expected flow of
information in high-quality reasoning chains. Finally, we show that scoring
reasoning chains based on ReCEval improves downstream task performance. Our
code is publicly available at: https://github.com/archiki/ReCEval
