  In this paper, we propose a novel text promptable surgical instrument
segmentation approach to overcome challenges associated with diversity and
differentiation of surgical instruments in minimally invasive surgeries. We
redefine the task as text promptable, thereby enabling a more nuanced
comprehension of surgical instruments and adaptability to new instrument types.
Inspired by recent advancements in vision-language models, we leverage
pretrained image and text encoders as our model backbone and design a text
promptable mask decoder consisting of attention- and convolution-based
prompting schemes for surgical instrument segmentation prediction. Our model
leverages multiple text prompts for each surgical instrument through a new
mixture of prompts mechanism, resulting in enhanced segmentation performance.
Additionally, we introduce a hard instrument area reinforcement module to
improve image feature comprehension and segmentation precision. Extensive
experiments on several surgical instrument segmentation datasets demonstrate
our model's superior performance and promising generalization capability. To
our knowledge, this is the first implementation of a promptable approach to
surgical instrument segmentation, offering significant potential for practical
application in the field of robotic-assisted surgery. Code is available at
https://github.com/franciszzj/TP-SIS.
