  Purpose: The recent Segment Anything Model (SAM) has demonstrated impressive
performance with point, text or bounding box prompts, in various applications.
However, in safety-critical surgical tasks, prompting is not possible due to
(i) the lack of per-frame prompts for supervised learning, (ii) it is
unrealistic to prompt frame-by-frame in a real-time tracking application, and
(iii) it is expensive to annotate prompts for offline applications.
  Methods: We develop Surgical-DeSAM to generate automatic bounding box prompts
for decoupling SAM to obtain instrument segmentation in real-time robotic
surgery. We utilise a commonly used detection architecture, DETR, and
fine-tuned it to obtain bounding box prompt for the instruments. We then
empolyed decoupling SAM (DeSAM) by replacing the image encoder with DETR
encoder and fine-tune prompt encoder and mask decoder to obtain instance
segmentation for the surgical instruments. To improve detection performance, we
adopted the Swin-transformer to better feature representation.
  Results: The proposed method has been validated on two publicly available
datasets from the MICCAI surgical instruments segmentation challenge EndoVis
2017 and 2018. The performance of our method is also compared with SOTA
instrument segmentation methods and demonstrated significant improvements with
dice metrics of 89.62 and 90.70 for the EndoVis 2017 and 2018.
  Conclusion: Our extensive experiments and validations demonstrate that
Surgical-DeSAM enables real-time instrument segmentation without any additional
prompting and outperforms other SOTA segmentation methods.
