  Common studies of gender bias in NLP focus either on extrinsic bias measured
by model performance on a downstream task or on intrinsic bias found in models'
internal representations. However, the relationship between extrinsic and
intrinsic bias is relatively unknown. In this work, we illuminate this
relationship by measuring both quantities together: we debias a model during
downstream fine-tuning, which reduces extrinsic bias, and measure the effect on
intrinsic bias, which is operationalized as bias extractability with
information-theoretic probing. Through experiments on two tasks and multiple
bias metrics, we show that our intrinsic bias metric is a better indicator of
debiasing than (a contextual adaptation of) the standard WEAT metric, and can
also expose cases of superficial debiasing. Our framework provides a
comprehensive perspective on bias in NLP models, which can be applied to deploy
NLP systems in a more informed manner. Our code and model checkpoints are
publicly available.
