  It is well-documented that word embeddings trained on large public corpora
consistently exhibit known human social biases. Although many methods for
debiasing exist, almost all fixate on completely eliminating biased information
from the embeddings and often diminish training set size in the process. In
this paper, we present a simple yet effective method for debiasing GloVe word
embeddings (Pennington et al., 2014) which works by incorporating explicit
information about training set bias rather than removing biased data outright.
Our method runs quickly and efficiently with the help of a fast bias gradient
approximation method from Brunet et al. (2019). As our approach is akin to the
notion of 'source criticism' in the humanities, we term our method
Source-Critical GloVe (SC-GloVe). We show that SC-GloVe reduces the effect size
on Word Embedding Association Test (WEAT) sets without sacrificing training
data or TOP-1 performance.
