  Misinformation emerges in times of uncertainty when credible information is
limited. This is challenging for NLP-based fact-checking as it relies on
counter-evidence, which may not yet be available. Despite increasing interest
in automatic fact-checking, it is still unclear if automated approaches can
realistically refute harmful real-world misinformation. Here, we contrast and
compare NLP fact-checking with how professional fact-checkers combat
misinformation in the absence of counter-evidence. In our analysis, we show
that, by design, existing NLP task definitions for fact-checking cannot refute
misinformation as professional fact-checkers do for the majority of claims. We
then define two requirements that the evidence in datasets must fulfill for
realistic fact-checking: It must be (1) sufficient to refute the claim and (2)
not leaked from existing fact-checking articles. We survey existing
fact-checking datasets and find that all of them fail to satisfy both criteria.
Finally, we perform experiments to demonstrate that models trained on a
large-scale fact-checking dataset rely on leaked evidence, which makes them
unsuitable in real-world scenarios. Taken together, we show that current NLP
fact-checking cannot realistically combat real-world misinformation because it
depends on unrealistic assumptions about counter-evidence in the data.
