  Deep learning could be prone to learning shortcuts raised by dataset bias and
result in inaccurate, unreliable, and unfair models, which impedes its adoption
in real-world clinical applications. Despite its significance, there is a
dearth of research in the medical image classification domain to address
dataset bias. Furthermore, the bias labels are often agnostic, as identifying
biases can be laborious and depend on post-hoc interpretation. This paper
proposes learning Adaptive Agreement from a Biased Council (Ada-ABC), a
debiasing framework that does not rely on explicit bias labels to tackle
dataset bias in medical images. Ada-ABC develops a biased council consisting of
multiple classifiers optimized with generalized cross entropy loss to learn the
dataset bias. A debiasing model is then simultaneously trained under the
guidance of the biased council. Specifically, the debiasing model is required
to learn adaptive agreement with the biased council by agreeing on the
correctly predicted samples and disagreeing on the wrongly predicted samples by
the biased council. In this way, the debiasing model could learn the target
attribute on the samples without spurious correlations while also avoiding
ignoring the rich information in samples with spurious correlations. We
theoretically demonstrated that the debiasing model could learn the target
features when the biased model successfully captures dataset bias. Moreover, to
our best knowledge, we constructed the first medical debiasing benchmark from
four datasets containing seven different bias scenarios. Our extensive
experiments practically showed that our proposed Ada-ABC outperformed
competitive approaches, verifying its effectiveness in mitigating dataset bias
for medical image classification. The codes and organized benchmark datasets
will be made publicly available.
