  We show that large pre-trained language models are inherently highly capable
of identifying label errors in natural language datasets: simply examining
out-of-sample data points in descending order of fine-tuned task loss
significantly outperforms more complex error-detection mechanisms proposed in
previous work.
  To this end, we contribute a novel method for introducing realistic,
human-originated label noise into existing crowdsourced datasets such as SNLI
and TweetNLP. We show that this noise has similar properties to real,
hand-verified label errors, and is harder to detect than existing synthetic
noise, creating challenges for model robustness. We argue that human-originated
noise is a better standard for evaluation than synthetic noise.
  Finally, we use crowdsourced verification to evaluate the detection of real
errors on IMDB, Amazon Reviews, and Recon, and confirm that pre-trained models
perform at a 9-36% higher absolute Area Under the Precision-Recall Curve than
existing models.
