  Segment Anything Model~(SAM), a prompt-driven foundation model for natural
image segmentation, has demonstrated impressive zero-shot performance. However,
SAM does not work when directly applied to medical image segmentation tasks,
since SAM lacks the functionality to predict semantic labels for predicted
masks and needs to provide extra prompts, such as points or boxes, to segment
target regions. Meanwhile, there is a huge gap between 2D natural images and 3D
medical images, so the performance of SAM is imperfect for medical image
segmentation tasks. Following the above issues, we propose MaskSAM, a novel
mask classification prompt-free SAM adaptation framework for medical image
segmentation. We design a prompt generator combined with the image encoder in
SAM to generate a set of auxiliary classifier tokens, auxiliary binary masks,
and auxiliary bounding boxes. Each pair of auxiliary mask and box prompts,
which can solve the requirements of extra prompts, is associated with class
label predictions by the sum of the auxiliary classifier token and the
learnable global classifier tokens in the mask decoder of SAM to solve the
predictions of semantic labels. Meanwhile, we design a 3D depth-convolution
adapter for image embeddings and a 3D depth-MLP adapter for prompt embeddings.
We inject one of them into each transformer block in the image encoder and mask
decoder to enable pre-trained 2D SAM models to extract 3D information and adapt
to 3D medical images. Our method achieves state-of-the-art performance on
AMOS2022, 90.52% Dice, which improved by 2.7% compared to nnUNet. Our method
surpasses nnUNet by 1.7% on ACDC and 1.0% on Synapse datasets.
