  Misinformation about critical issues such as climate change and vaccine
safety is oftentimes amplified on online social and search platforms. The
crowdsourcing of content credibility assessment by laypeople has been proposed
as one strategy to combat misinformation by attempting to replicate the
assessments of experts at scale. In this work, we investigate news credibility
assessments by crowds versus experts to understand when and how ratings between
them differ. We gather a dataset of over 4,000 credibility assessments taken
from 2 crowd groups---journalism students and Upwork workers---as well as 2
expert groups---journalists and scientists---on a varied set of 50 news
articles related to climate science, a topic with widespread disconnect between
public opinion and expert consensus. Examining the ratings, we find differences
in performance due to the makeup of the crowd, such as rater demographics and
political leaning, as well as the scope of the tasks that the crowd is assigned
to rate, such as the genre of the article and partisanship of the publication.
Finally, we find differences between expert assessments due to differing expert
criteria that journalism versus science experts use---differences that may
contribute to crowd discrepancies, but that also suggest a way to reduce the
gap by designing crowd tasks tailored to specific expert criteria. From these
findings, we outline future research directions to better design crowd
processes that are tailored to specific crowds and types of content.
