  Few-shot segmentation performance declines substantially when facing images
from a domain different than the training domain, effectively limiting
real-world use cases. To alleviate this, recently cross-domain few-shot
segmentation (CD-FSS) has emerged. Works that address this task mainly
attempted to learn segmentation on a source domain in a manner that generalizes
across domains. Surprisingly, we can outperform these approaches while
eliminating the training stage and removing their main segmentation network. We
show test-time task-adaption is the key for successful CD-FSS instead.
Task-adaption is achieved by appending small networks to the feature pyramid of
a conventionally classification-pretrained backbone. To avoid overfitting to
the few labeled samples in supervised fine-tuning, consistency across augmented
views of input images serves as guidance while learning the parameters of the
attached layers. Despite our self-restriction not to use any images other than
the few labeled samples at test time, we achieve new state-of-the-art
performance in CD-FSS, evidencing the need to rethink approaches for the task.
