  Hate speech classifiers trained on imbalanced datasets struggle to determine
if group identifiers like "gay" or "black" are used in offensive or prejudiced
ways. Such biases manifest in false positives when these identifiers are
present, due to models' inability to learn the contexts which constitute a
hateful usage of identifiers. We extract SOC post-hoc explanations from
fine-tuned BERT classifiers to efficiently detect bias towards identity terms.
Then, we propose a novel regularization technique based on these explanations
that encourages models to learn from the context of group identifiers in
addition to the identifiers themselves. Our approach improved over baselines in
limiting false positives on out-of-domain data while maintaining or improving
in-domain performance. Project page:
https://inklab.usc.edu/contextualize-hate-speech/.
