  Truthfulness judgments are a fundamental step in the process of fighting
misinformation, as they are crucial to train and evaluate classifiers that
automatically distinguish true and false statements. Usually such judgments are
made by experts, like journalists for political statements or medical doctors
for medical statements. In this paper, we follow a different approach and rely
on (non-expert) crowd workers. This of course leads to the following research
question: Can crowdsourcing be reliably used to assess the truthfulness of
information and to create large-scale labeled collections for information
credibility systems? To address this issue, we present the results of an
extensive study based on crowdsourcing: we collect thousands of truthfulness
assessments over two datasets, and we compare expert judgments with crowd
judgments, expressed on scales with various granularity levels. We also measure
the political bias and the cognitive background of the workers, and quantify
their effect on the reliability of the data provided by the crowd.
