  Existing datasets for automated fact-checking have substantial limitations,
such as relying on artificial claims, lacking annotations for evidence and
intermediate reasoning, or including evidence published after the claim. In
this paper we introduce AVeriTeC, a new dataset of 4,568 real-world claims
covering fact-checks by 50 different organizations. Each claim is annotated
with question-answer pairs supported by evidence available online, as well as
textual justifications explaining how the evidence combines to produce a
verdict. Through a multi-round annotation process, we avoid common pitfalls
including context dependence, evidence insufficiency, and temporal leakage, and
reach a substantial inter-annotator agreement of $\kappa=0.619$ on verdicts. We
develop a baseline as well as an evaluation scheme for verifying claims through
several question-answering steps against the open web.
