  Debiasing methods in NLP models traditionally focus on isolating information
related to a sensitive attribute (e.g., gender or race). We instead argue that
a favorable debiasing method should use sensitive information 'fairly,' with
explanations, rather than blindly eliminating it. This fair balance is often
subjective and can be challenging to achieve algorithmically. We explore two
interactive setups with a frozen predictive model and show that users able to
provide feedback can achieve a better and fairer balance between task
performance and bias mitigation. In one setup, users, by interacting with test
examples, further decreased bias in the explanations (5-8%) while maintaining
the same prediction accuracy. In the other setup, human feedback was able to
disentangle associated bias and predictive information from the input leading
to superior bias mitigation and improved task performance (4-5%)
simultaneously.
