  Polyp segmentation is a crucial step towards computer-aided diagnosis of
colorectal cancer. However, most of the polyp segmentation methods require
pixel-wise annotated datasets. Annotated datasets are tedious and
time-consuming to produce, especially for physicians who must dedicate their
time to their patients. We tackle this issue by proposing a novel framework
that can be trained using only weakly annotated images along with exploiting
unlabeled images. To this end, we propose three ideas to address this problem,
more specifically our contributions are: 1) a novel sparse foreground loss that
suppresses false positives and improves weakly-supervised training, 2) a
batch-wise weighted consistency loss utilizing predicted segmentation maps from
identical networks trained using different initialization during
semi-supervised training, 3) a deformable transformer encoder neck for feature
enhancement by fusing information across levels and flexible spatial locations.
  Extensive experimental results demonstrate the merits of our ideas on five
challenging datasets outperforming some state-of-the-art fully supervised
models. Also, our framework can be utilized to fine-tune models trained on
natural image segmentation datasets drastically improving their performance for
polyp segmentation and impressively demonstrating superior performance to fully
supervised fine-tuning.
