  Deep neural networks (DNNs) trained on one set of medical images often
experience severe performance drop on unseen test images, due to various domain
discrepancy between the training images (source domain) and the test images
(target domain), which raises a domain adaptation issue. In clinical settings,
it is difficult to collect enough annotated target domain data in a short
period. Few-shot domain adaptation, i.e., adapting a trained model with a
handful of annotations, is highly practical and useful in this case. In this
paper, we propose a Polymorphic Transformer (Polyformer), which can be
incorporated into any DNN backbones for few-shot domain adaptation.
Specifically, after the polyformer layer is inserted into a model trained on
the source domain, it extracts a set of prototype embeddings, which can be
viewed as a "basis" of the source-domain features. On the target domain, the
polyformer layer adapts by only updating a projection layer which controls the
interactions between image features and the prototype embeddings. All other
model weights (except BatchNorm parameters) are frozen during adaptation. Thus,
the chance of overfitting the annotations is greatly reduced, and the model can
perform robustly on the target domain after being trained on a few annotated
images. We demonstrate the effectiveness of Polyformer on two medical
segmentation tasks (i.e., optic disc/cup segmentation, and polyp segmentation).
The source code of Polyformer is released at
https://github.com/askerlee/segtran.
