  Colonoscopy videos provide richer information in polyp segmentation for
rectal cancer diagnosis. However, the endoscope's fast moving and close-up
observing make the current methods suffer from large spatial incoherence and
continuous low-quality frames, and thus yield limited segmentation accuracy. In
this context, we focus on robust video polyp segmentation by enhancing the
adjacent feature consistency and rebuilding the reliable polyp representation.
To achieve this goal, we in this paper propose SALI network, a hybrid of
Short-term Alignment Module (SAM) and Long-term Interaction Module (LIM). The
SAM learns spatial-aligned features of adjacent frames via deformable
convolution and further harmonizes them to capture more stable short-term polyp
representation. In case of low-quality frames, the LIM stores the historical
polyp representations as a long-term memory bank, and explores the
retrospective relations to interactively rebuild more reliable polyp features
for the current segmentation. Combing SAM and LIM, the SALI network of video
segmentation shows a great robustness to the spatial variations and low-visual
cues. Benchmark on the large-scale SUNSEG verifies the superiority of SALI over
the current state-of-the-arts by improving Dice by 2.1%, 2.5%, 4.1% and 1.9%,
for the four test sub-sets, respectively. Codes are at
https://github.com/Scatteredrain/SALI.
