  Ensuring fairness in deep-learning-based segmentors is crucial for health
equity. Much effort has been dedicated to mitigating unfairness in the training
datasets or procedures. However, with the increasing prevalence of foundation
models in medical image analysis, it is hard to train fair models from scratch
while preserving utility. In this paper, we propose a novel method, Adversarial
Privacy-aware Perturbations on Latent Embedding (APPLE), that can improve the
fairness of deployed segmentors by introducing a small latent feature perturber
without updating the weights of the original model. By adding perturbation to
the latent vector, APPLE decorates the latent vector of segmentors such that no
fairness-related features can be passed to the decoder of the segmentors while
preserving the architecture and parameters of the segmentor. Experiments on two
segmentation datasets and five segmentors (three U-Net-like and two SAM-like)
illustrate the effectiveness of our proposed method compared to several
unfairness mitigation methods.
