  Recently, the misinformation problem has been addressed with a
crowdsourcing-based approach: to assess the truthfulness of a statement,
instead of relying on a few experts, a crowd of non-expert is exploited. We
study whether crowdsourcing is an effective and reliable method to assess
truthfulness during a pandemic, targeting statements related to COVID-19, thus
addressing (mis)information that is both related to a sensitive and personal
issue and very recent as compared to when the judgment is done. In our
experiments, crowd workers are asked to assess the truthfulness of statements,
and to provide evidence for the assessments. Besides showing that the crowd is
able to accurately judge the truthfulness of the statements, we report results
on workers behavior, agreement among workers, effect of aggregation functions,
of scales transformations, and of workers background and bias. We perform a
longitudinal study by re-launching the task multiple times with both novice and
experienced workers, deriving important insights on how the behavior and
quality change over time. Our results show that: workers are able to detect and
objectively categorize online (mis)information related to COVID-19; both
crowdsourced and expert judgments can be transformed and aggregated to improve
quality; worker background and other signals (e.g., source of information,
behavior) impact the quality of the data. The longitudinal study demonstrates
that the time-span has a major effect on the quality of the judgments, for both
novice and experienced workers. Finally, we provide an extensive failure
analysis of the statements misjudged by the crowd-workers.
