  The Segment Anything Model (SAM) is a powerful vision foundation model that
is revolutionizing the traditional paradigm of segmentation. Despite this, a
reliance on prompting each frame and large computational cost limit its usage
in robotically assisted surgery. Applications, such as augmented reality
guidance, require little user intervention along with efficient inference to be
usable clinically. In this study, we address these limitations by adopting
lightweight SAM variants to meet the efficiency requirement and employing
fine-tuning techniques to enhance their generalization in surgical scenes.
Recent advancements in Tracking Any Point (TAP) have shown promising results in
both accuracy and efficiency, particularly when points are occluded or leave
the field of view. Inspired by this progress, we present a novel framework that
combines an online point tracker with a lightweight SAM model that is
fine-tuned for surgical instrument segmentation. Sparse points within the
region of interest are tracked and used to prompt SAM throughout the video
sequence, providing temporal consistency. The quantitative results surpass the
state-of-the-art semi-supervised video object segmentation method XMem on the
EndoVis 2015 dataset with 84.8 IoU and 91.0 Dice. Our method achieves promising
performance that is comparable to XMem and transformer-based fully supervised
segmentation methods on ex vivo UCL dVRK and in vivo CholecSeg8k datasets. In
addition, the proposed method shows promising zero-shot generalization ability
on the label-free STIR dataset. In terms of efficiency, we tested our method on
a single GeForce RTX 4060/4090 GPU respectively, achieving an over 25/90 FPS
inference speed. Code is available at:
https://github.com/wuzijian1997/SIS-PT-SAM
