  In the era of Foundation Models' (FMs) rising prominence in AI, our study
addresses the challenge of biases in medical images while the model operates in
black-box (e.g., using FM API), particularly spurious correlations between
pixels and sensitive attributes. Traditional methods for bias mitigation face
limitations due to the restricted access to web-hosted FMs and difficulties in
addressing the underlying bias encoded within the FM API. We propose a
D(ebiased) N(oise) E(diting) strategy, termed DNE, which generates DNE noise to
mask such spurious correlation. DNE is capable of mitigating bias both within
the FM API embedding and the images themselves. Furthermore, DNE is suitable
for both white-box and black-box FM APIs, where we introduced G(reedy)
(Z)eroth-O(rder) (GeZO) optimization for it when the gradient is inaccessible
in black-box APIs. Our whole pipeline enables fairness-aware image editing that
can be applied across various medical contexts without requiring direct model
manipulation or significant computational resources. Our empirical results
demonstrate the method's effectiveness in maintaining fairness and utility
across different patient groups and diseases. In the era of AI-driven medicine,
this work contributes to making healthcare diagnostics more equitable,
showcasing a practical solution for bias mitigation in pre-trained image FMs.
Our code is provided at
https://github.com/ubc-tea/DNE-foundation-model-fairness.
