  BERT-based models have had strong performance on leaderboards, yet have been
demonstrably worse in real-world settings requiring generalization. Limited
quantities of training data is considered a key impediment to achieving
generalizability in machine learning. In this paper, we examine the impact of
training data quality, not quantity, on a model's generalizability. We consider
two characteristics of training data: the portion of human-adversarial
(h-adversarial), i.e., sample pairs with seemingly minor differences but
different ground-truth labels, and human-affable (h-affable) training samples,
i.e., sample pairs with minor differences but the same ground-truth label. We
find that for a fixed size of training samples, as a rule of thumb, having
10-30% h-adversarial instances improves the precision, and therefore F1, by up
to 20 points in the tasks of text classification and relation extraction.
Increasing h-adversarials beyond this range can result in performance plateaus
or even degradation. In contrast, h-affables may not contribute to a model's
generalizability and may even degrade generalization performance.
