  Prompting language models to provide step-by-step answers (e.g.,
"Chain-of-Thought") is the prominent approach for complex reasoning tasks,
where more accurate reasoning chains typically improve downstream task
performance. Recent literature discusses automatic methods to verify reasoning
to evaluate and improve their correctness. However, no fine-grained step-level
datasets are available to enable thorough evaluation of such verification
methods, hindering progress in this direction. We introduce REVEAL: Reasoning
Verification Evaluation, a dataset to benchmark automatic verifiers of complex
Chain-of-Thought reasoning in open-domain question-answering settings. REVEAL
includes comprehensive labels for the relevance, attribution to evidence
passages, and logical correctness of each reasoning step in a language model's
answer, across a variety of datasets and state-of-the-art language models.
Evaluation on REVEAL shows that verifiers struggle at verifying reasoning
chains - in particular, verifying logical correctness and detecting
contradictions. Available at https://reveal-dataset.github.io/ .
