  Many works have shown that deep learning-based medical image classification
models can exhibit bias toward certain demographic attributes like race,
gender, and age. Existing bias mitigation methods primarily focus on learning
debiased models, which may not necessarily guarantee all sensitive information
can be removed and usually comes with considerable accuracy degradation on both
privileged and unprivileged groups. To tackle this issue, we propose a method,
FairPrune, that achieves fairness by pruning. Conventionally, pruning is used
to reduce the model size for efficient inference. However, we show that pruning
can also be a powerful tool to achieve fairness. Our observation is that during
pruning, each parameter in the model has different importance for different
groups' accuracy. By pruning the parameters based on this importance
difference, we can reduce the accuracy difference between the privileged group
and the unprivileged group to improve fairness without a large accuracy drop.
To this end, we use the second derivative of the parameters of a pre-trained
model to quantify the importance of each parameter with respect to the model
accuracy for each group. Experiments on two skin lesion diagnosis datasets over
multiple sensitive attributes demonstrate that our method can greatly improve
fairness while keeping the average accuracy of both groups as high as possible.
