  We present a human-in-the-loop evaluation framework for fact-checking novel
misinformation claims and identifying social media messages that support them.
Our approach extracts check-worthy claims, which are aggregated and ranked for
review. Stance classifiers are then used to identify tweets supporting novel
misinformation claims, which are further reviewed to determine whether they
violate relevant policies. To demonstrate the feasibility of our approach, we
develop a baseline system based on modern NLP methods for human-in-the-loop
fact-checking in the domain of COVID-19 treatments. We make our data and
detailed annotation guidelines available to support the evaluation of
human-in-the-loop systems that identify novel misinformation directly from raw
user-generated content.
