  We present a new fact-checking benchmark, Check-COVID, that requires systems
to verify claims about COVID-19 from news using evidence from scientific
articles. This approach to fact-checking is particularly challenging as it
requires checking internet text written in everyday language against evidence
from journal articles written in formal academic language. Check-COVID contains
1, 504 expert-annotated news claims about the coronavirus paired with
sentence-level evidence from scientific journal articles and veracity labels.
It includes both extracted (journalist-written) and composed
(annotator-written) claims. Experiments using both a fact-checking specific
system and GPT-3.5, which respectively achieve F1 scores of 76.99 and 69.90 on
this task, reveal the difficulty of automatically fact-checking both claim
types and the importance of in-domain data for good performance. Our data and
models are released publicly at https://github.com/posuer/Check-COVID.
