  Segment Anything Model (SAM) has received remarkable attention as it offers a
powerful and versatile solution for object segmentation in images. However,
fine-tuning SAM for downstream segmentation tasks under different scenarios
remains a challenge, as the varied characteristics of different scenarios
naturally requires diverse model parameter spaces. Most existing fine-tuning
methods attempt to bridge the gaps among different scenarios by introducing a
set of new parameters to modify SAM's original parameter space. Unlike these
works, in this paper, we propose fine-tuning SAM efficiently by parameter space
reconstruction (SAM-PARSER), which introduce nearly zero trainable parameters
during fine-tuning. In SAM-PARSER, we assume that SAM's original parameter
space is relatively complete, so that its bases are able to reconstruct the
parameter space of a new scenario. We obtain the bases by matrix decomposition,
and fine-tuning the coefficients to reconstruct the parameter space tailored to
the new scenario by an optimal linear combination of the bases. Experimental
results show that SAM-PARSER exhibits superior segmentation performance across
various scenarios, while reducing the number of trainable parameters by
$\approx 290$ times compared with current parameter-efficient fine-tuning
methods.
