  Discrimination has been shown in many machine learning applications, which
calls for sufficient fairness testing before their deployment in ethic-relevant
domains such as face recognition, medical diagnosis and criminal sentence.
Existing fairness testing approaches are mostly designed for identifying
individual discrimination, i.e., discrimination against individuals. Yet, as
another widely concerning type of discrimination, testing against group
discrimination, mostly hidden, is much less studied. To address the gap, in
this work, we propose TESTSGD, an interpretable testing approach which
systematically identifies and measures hidden (which we call `subtle' group
discrimination} of a neural network characterized by conditions over
combinations of the sensitive features. Specifically, given a neural network,
TESTSGDfirst automatically generates an interpretable rule set which
categorizes the input space into two groups exposing the model's group
discrimination. Alongside, TESTSGDalso provides an estimated group fairness
score based on sampling the input space to measure the degree of the identified
subtle group discrimination, which is guaranteed to be accurate up to an error
bound. We evaluate TESTSGDon multiple neural network models trained on popular
datasets including both structured data and text data. The experiment results
show that TESTSGDis effective and efficient in identifying and measuring such
subtle group discrimination that has never been revealed before. Furthermore,
we show that the testing results of TESTSGDcan guide generation of new samples
to mitigate such discrimination through retraining with negligible accuracy
drop.
