  Polyp segmentation plays a pivotal role in colorectal cancer diagnosis.
Recently, the emergence of the Segment Anything Model (SAM) has introduced
unprecedented potential for polyp segmentation, leveraging its powerful
pre-training capability on large-scale datasets. However, due to the domain gap
between natural and endoscopy images, SAM encounters two limitations in
achieving effective performance in polyp segmentation. Firstly, its
Transformer-based structure prioritizes global and low-frequency information,
potentially overlooking local details, and introducing bias into the learned
features. Secondly, when applied to endoscopy images, its poor
out-of-distribution (OOD) performance results in substandard predictions and
biased confidence output. To tackle these challenges, we introduce a novel
approach named Augmented SAM for Polyp Segmentation (ASPS), equipped with two
modules: Cross-branch Feature Augmentation (CFA) and Uncertainty-guided
Prediction Regularization (UPR). CFA integrates a trainable CNN encoder branch
with a frozen ViT encoder, enabling the integration of domain-specific
knowledge while enhancing local features and high-frequency details. Moreover,
UPR ingeniously leverages SAM's IoU score to mitigate uncertainty during the
training procedure, thereby improving OOD performance and domain
generalization. Extensive experimental results demonstrate the effectiveness
and utility of the proposed method in improving SAM's performance in polyp
segmentation. Our code is available at https://github.com/HuiqianLi/ASPS.
