  Freezing the pre-trained backbone has become a standard paradigm to avoid
overfitting in few-shot segmentation. In this paper, we rethink the paradigm
and explore a new regime: {\em fine-tuning a small part of parameters in the
backbone}. We present a solution to overcome the overfitting problem, leading
to better model generalization on learning novel classes. Our method decomposes
backbone parameters into three successive matrices via the Singular Value
Decomposition (SVD), then {\em only fine-tunes the singular values} and keeps
others frozen. The above design allows the model to adjust feature
representations on novel classes while maintaining semantic clues within the
pre-trained backbone. We evaluate our {\em Singular Value Fine-tuning (SVF)}
approach on various few-shot segmentation methods with different backbones. We
achieve state-of-the-art results on both Pascal-5$^i$ and COCO-20$^i$ across
1-shot and 5-shot settings. Hopefully, this simple baseline will encourage
researchers to rethink the role of backbone fine-tuning in few-shot settings.
The source code and models will be available at https://github.com/syp2ysy/SVF.
