  The recently introduced Segment-Anything Model (SAM) has the potential to
greatly accelerate the development of segmentation models. However, directly
applying SAM to surgical images has key limitations including (1) the
requirement of image-specific prompts at test-time, thereby preventing fully
automated segmentation, and (2) ineffectiveness due to substantial domain gap
between natural and surgical images. In this work, we propose CycleSAM, an
approach for one-shot surgical scene segmentation that uses the training
image-mask pair at test-time to automatically identify points in the test
images that correspond to each object class, which can then be used to prompt
SAM to produce object masks. To produce high-fidelity matches, we introduce a
novel spatial cycle-consistency constraint that enforces point proposals in the
test image to rematch to points within the object foreground region in the
training image. Then, to address the domain gap, rather than directly using the
visual features from SAM, we employ a ResNet50 encoder pretrained on surgical
images in a self-supervised fashion, thereby maintaining high label-efficiency.
We evaluate CycleSAM for one-shot segmentation on two diverse surgical semantic
segmentation datasets, comprehensively outperforming baseline approaches and
reaching up to 50% of fully-supervised performance.
