  Gender bias in language models has attracted sufficient attention because it
threatens social justice. However, most of the current debiasing methods
degraded the model's performance on other tasks while the degradation mechanism
is still mysterious. We propose a theoretical framework explaining the three
candidate mechanisms of the language model's gender bias. We use our
theoretical framework to explain why the current debiasing methods cause
performance degradation. We also discover a pathway through which debiasing
will not degrade the model performance. We further develop a
causality-detection fine-tuning approach to correct gender bias. The numerical
experiment demonstrates that our method is able to lead to double dividends:
partially mitigating gender bias while avoiding performance degradation.
