  We propose a model-agnostic approach for mitigating the prediction bias of a
black-box decision-maker, and in particular, a human decision-maker. Our method
detects in the feature space where the black-box decision-maker is biased and
replaces it with a few short decision rules, acting as a "fair surrogate". The
rule-based surrogate model is trained under two objectives, predictive
performance and fairness. Our model focuses on a setting that is common in
practice but distinct from other literature on fairness. We only have black-box
access to the model, and only a limited set of true labels can be queried under
a budget constraint. We formulate a multi-objective optimization for building a
surrogate model, where we simultaneously optimize for both predictive
performance and bias. To train the model, we propose a novel training algorithm
that combines a nondominated sorting genetic algorithm with active learning. We
test our model on public datasets where we simulate various biased "black-box"
classifiers (decision-makers) and apply our approach for interpretable
augmented fairness.
