  Deep learning has thrived by training on large-scale datasets. However, in
many applications, as for medical image diagnosis, getting massive amount of
data is still prohibitive due to privacy, lack of acquisition homogeneity and
annotation cost. In this scenario, transfer learning from natural image
collections is a standard practice that attempts to tackle shape, texture and
color discrepancies all at once through pretrained model fine-tuning. In this
work, we propose to disentangle those challenges and design a dedicated network
module that focuses on color adaptation. We combine learning from scratch of
the color module with transfer learning of different classification backbones,
obtaining an end-to-end, easy-to-train architecture for diagnostic image
recognition on X-ray images. Extensive experiments showed how our approach is
particularly efficient in case of data scarcity and provides a new path for
further transferring the learned color information across multiple medical
datasets.
