  Language representations are known to carry stereotypical biases and, as a
result, lead to biased predictions in downstream tasks. While existing methods
are effective at mitigating biases by linear projection, such methods are too
aggressive: they not only remove bias, but also erase valuable information from
word embeddings. We develop new measures for evaluating specific information
retention that demonstrate the tradeoff between bias removal and information
retention. To address this challenge, we propose OSCaR (Orthogonal Subspace
Correction and Rectification), a bias-mitigating method that focuses on
disentangling biased associations between concepts instead of removing concepts
wholesale. Our experiments on gender biases show that OSCaR is a well-balanced
approach that ensures that semantic information is retained in the embeddings
and bias is also effectively mitigated.
