  Automated fact-checking systems verify claims against evidence to predict
their veracity. In real-world scenarios, the retrieved evidence may not
unambiguously support or refute the claim and yield conflicting but valid
interpretations. Existing fact-checking datasets assume that the models
developed with them predict a single veracity label for each claim, thus
discouraging the handling of such ambiguity. To address this issue we present
AmbiFC, a fact-checking dataset with 10k claims derived from real-world
information needs. It contains fine-grained evidence annotations of 50k
passages from 5k Wikipedia pages. We analyze the disagreements arising from
ambiguity when comparing claims against evidence in AmbiFC, observing a strong
correlation of annotator disagreement with linguistic phenomena such as
underspecification and probabilistic reasoning. We develop models for
predicting veracity handling this ambiguity via soft labels and find that a
pipeline that learns the label distribution for sentence-level evidence
selection and veracity prediction yields the best performance. We compare
models trained on different subsets of AmbiFC and show that models trained on
the ambiguous instances perform better when faced with the identified
linguistic phenomena.
