  Fairness has been a critical issue that affects the adoption of deep learning
models in real practice. To improve model fairness, many existing methods have
been proposed and evaluated to be effective in their own contexts. However,
there is still no systematic evaluation among them for a comprehensive
comparison under the same context, which makes it hard to understand the
performance distinction among them, hindering the research progress and
practical adoption of them. To fill this gap, this paper endeavours to conduct
the first large-scale empirical study to comprehensively compare the
performance of existing state-of-the-art fairness improving techniques.
Specifically, we target the widely-used application scenario of image
classification, and utilized three different datasets and five commonly-used
performance metrics to assess in total 13 methods from diverse categories. Our
findings reveal substantial variations in the performance of each method across
different datasets and sensitive attributes, indicating over-fitting on
specific datasets by many existing methods. Furthermore, different fairness
evaluation metrics, due to their distinct focuses, yield significantly
different assessment results. Overall, we observe that pre-processing methods
and in-processing methods outperform post-processing methods, with
pre-processing methods exhibiting the best performance. Our empirical study
offers comprehensive recommendations for enhancing fairness in deep learning
models. We approach the problem from multiple dimensions, aiming to provide a
uniform evaluation platform and inspire researchers to explore more effective
fairness solutions via a set of implications.
