  Skin lesion analysis models are biased by artifacts placed during image
acquisition, which influence model predictions despite carrying no clinical
information. Solutions that address this problem by regularizing models to
prevent learning those spurious features achieve only partial success, and
existing test-time debiasing techniques are inappropriate for skin lesion
analysis due to either making unrealistic assumptions on the distribution of
test data or requiring laborious annotation from medical practitioners. We
propose TTS (Test-Time Selection), a human-in-the-loop method that leverages
positive (e.g., lesion area) and negative (e.g., artifacts) keypoints in test
samples. TTS effectively steers models away from exploiting spurious
artifact-related correlations without retraining, and with less annotation
requirements. Our solution is robust to a varying availability of annotations,
and different levels of bias. We showcase on the ISIC2019 dataset (for which we
release a subset of annotated images) how our model could be deployed in the
real-world for mitigating bias.
