  Modern NLP systems exhibit a range of biases, which a growing literature on
model debiasing attempts to correct. However current progress is hampered by a
plurality of definitions of bias, means of quantification, and oftentimes vague
relation between debiasing algorithms and theoretical measures of bias. This
paper seeks to clarify the current situation and plot a course for meaningful
progress in fair learning, with two key contributions: (1) making clear
inter-relations among the current gamut of methods, and their relation to
fairness theory; and (2) addressing the practical problem of model selection,
which involves a trade-off between fairness and accuracy and has led to
systemic issues in fairness research. Putting them together, we make several
recommendations to help shape future work.
