  We examine the disconnect between scholarship and practice in applying
machine learning to trust and safety problems, using misinformation detection
as a case study. We survey literature on automated detection of misinformation
across a corpus of 248 well-cited papers in the field. We then examine subsets
of papers for data and code availability, design missteps, reproducibility, and
generalizability. Our paper corpus includes published work in security, natural
language processing, and computational social science. Across these disparate
disciplines, we identify common errors in dataset and method design. In
general, detection tasks are often meaningfully distinct from the challenges
that online services actually face. Datasets and model evaluation are often
non-representative of real-world contexts, and evaluation frequently is not
independent of model training. We demonstrate the limitations of current
detection methods in a series of three representative replication studies.
Based on the results of these analyses and our literature survey, we conclude
that the current state-of-the-art in fully-automated misinformation detection
has limited efficacy in detecting human-generated misinformation. We offer
recommendations for evaluating applications of machine learning to trust and
safety problems and recommend future directions for research.
