  Debiasing language models from unwanted behaviors in Natural Language
Understanding tasks is a topic with rapidly increasing interest in the NLP
community. Spurious statistical correlations in the data allow models to
perform shortcuts and avoid uncovering more advanced and desirable linguistic
features. A multitude of effective debiasing approaches has been proposed, but
flexibility remains a major issue. For the most part, models must be retrained
to find a new set of weights with debiased behavior. We propose a new debiasing
method in which we identify debiased pruning masks that can be applied to a
finetuned model. This enables the selective and conditional application of
debiasing behaviors. We assume that bias is caused by a certain subset of
weights in the network; our method is, in essence, a mask search to identify
and remove biased weights. Our masks show equivalent or superior performance to
the standard counterparts, while offering important benefits. Pruning masks can
be stored with high efficiency in memory, and it becomes possible to switch
among several debiasing behaviors (or revert back to the original biased model)
at inference time. Finally, it opens the doors to further research on how
biases are acquired by studying the generated masks. For example, we observed
that the early layers and attention heads were pruned more aggressively,
possibly hinting towards the location in which biases may be encoded.
