  In this paper we introduce a new publicly available dataset for verification
against textual sources, FEVER: Fact Extraction and VERification. It consists
of 185,445 claims generated by altering sentences extracted from Wikipedia and
subsequently verified without knowledge of the sentence they were derived from.
The claims are classified as Supported, Refuted or NotEnoughInfo by annotators
achieving 0.6841 in Fleiss $\kappa$. For the first two classes, the annotators
also recorded the sentence(s) forming the necessary evidence for their
judgment. To characterize the challenge of the dataset presented, we develop a
pipeline approach and compare it to suitably designed oracles. The best
accuracy we achieve on labeling a claim accompanied by the correct evidence is
31.87%, while if we ignore the evidence we achieve 50.91%. Thus we believe that
FEVER is a challenging testbed that will help stimulate progress on claim
verification against textual sources.
