  Driven by the latest trend towards self-supervised learning (SSL), the
paradigm of "pretraining-then-finetuning" has been extensively explored to
enhance the performance of clinical applications with limited annotations.
Previous literature on model finetuning has mainly focused on regularization
terms and specific policy models, while the misalignment of channels between
source and target models has not received sufficient attention. In this work,
we revisited the dynamics of batch normalization (BN) layers and observed that
the trainable affine parameters of BN serve as sensitive indicators of domain
information. Therefore, Affine Collaborative Normalization (AC-Norm) is
proposed for finetuning, which dynamically recalibrates the channels in the
target model according to the cross-domain channel-wise correlations without
adding extra parameters. Based on a single-step backpropagation, AC-Norm can
also be utilized to measure the transferability of pretrained models. We
evaluated AC-Norm against the vanilla finetuning and state-of-the-art
fine-tuning methods on transferring diverse pretrained models to the diabetic
retinopathy grade classification, retinal vessel segmentation, CT lung nodule
segmentation/classification, CT liver-tumor segmentation and MRI cardiac
segmentation tasks. Extensive experiments demonstrate that AC-Norm unanimously
outperforms the vanilla finetuning by up to 4% improvement, even under
significant domain shifts where the state-of-the-art methods bring no gains. We
also prove the capability of AC-Norm in fast transferability estimation. Our
code is available at https://github.com/EndoluminalSurgicalVision-IMR/ACNorm.
