  Segment anything model (SAM) demonstrates strong generalization ability on
natural image segmentation. However, its direct adaption in medical image
segmentation tasks shows significant performance drops with inferior accuracy
and unstable results. It may also requires an excessive number of prompt points
to obtain a reasonable accuracy. For segmenting 3D radiological CT or MRI
scans, a 2D SAM model has to separately handle hundreds of 2D slices. Although
quite a few studies explore adapting SAM into medical image volumes, the
efficiency of 2D adaption methods is unsatisfactory and 3D adaptation methods
only capable of segmenting specific organs/tumors. In this work, we propose a
comprehensive and scalable 3D SAM model for whole-body CT segmentation, named
CT-SAM3D. Instead of adapting SAM, we propose a 3D promptable segmentation
model using a (nearly) fully labeled CT dataset. To train CT-SAM3D effectively,
ensuring the model's accurate responses to higher-dimensional spatial prompts
is crucial, and 3D patch-wise training is required due to GPU memory
constraints. For this purpose, we propose two key technical developments: 1) a
progressively and spatially aligned prompt encoding method to effectively
encode click prompts in local 3D space; and 2) a cross-patch prompt learning
scheme to capture more 3D spatial context, which is beneficial for reducing the
editing workloads when interactively prompting on large organs. CT-SAM3D is
trained and validated using a curated dataset of 1204 CT scans containing 107
whole-body anatomies, reporting significantly better quantitative performance
against all previous SAM-derived models by a large margin with much fewer click
prompts. Our model can handle segmenting unseen organ as well. Code, data, and
our 3D interactive segmentation tool with quasi-real-time responses will be
made publicly available.
