  Machine learning models are vulnerable to biases that result in unfair
treatment of individuals from different populations. Recent work that aims to
test a model's fairness at the individual level either relies on domain
knowledge to choose metrics, or on input transformations that risk generating
out-of-domain samples. We describe a new approach for testing individual
fairness that does not have either requirement. We propose a novel criterion
for evaluating individual fairness and develop a practical testing method based
on this criterion which we call fAux (pronounced fox). This is based on
comparing the derivatives of the predictions of the model to be tested with
those of an auxiliary model, which predicts the protected variable from the
observed data. We show that the proposed method effectively identifies
discrimination on both synthetic and real-world datasets, and has quantitative
and qualitative advantages over contemporary methods.
