  Fairness has become increasingly pivotal in medical image recognition.
However, without mitigating bias, deploying unfair medical AI systems could
harm the interests of underprivileged populations. In this paper, we observe
that while features extracted from the deeper layers of neural networks
generally offer higher accuracy, fairness conditions deteriorate as we extract
features from deeper layers. This phenomenon motivates us to extend the concept
of multi-exit frameworks. Unlike existing works mainly focusing on accuracy,
our multi-exit framework is fairness-oriented; the internal classifiers are
trained to be more accurate and fairer, with high extensibility to apply to
most existing fairness-aware frameworks. During inference, any instance with
high confidence from an internal classifier is allowed to exit early.
Experimental results show that the proposed framework can improve the fairness
condition over the state-of-the-art in two dermatological disease datasets.
