  Accurate polyp detection is essential for assisting clinical rectal cancer
diagnoses. Colonoscopy videos contain richer information than still images,
making them a valuable resource for deep learning methods. Great efforts have
been made to conduct video polyp detection through multi-frame temporal/spatial
aggregation. However, unlike common fixed-camera video, the camera-moving scene
in colonoscopy videos can cause rapid video jitters, leading to unstable
training for existing video detection models. Additionally, the concealed
nature of some polyps and the complex background environment further hinder the
performance of existing video detectors. In this paper, we propose the
\textbf{YONA} (\textbf{Y}ou \textbf{O}nly \textbf{N}eed one \textbf{A}djacent
Reference-frame) method, an efficient end-to-end training framework for video
polyp detection. YONA fully exploits the information of one previous adjacent
frame and conducts polyp detection on the current frame without multi-frame
collaborations. Specifically, for the foreground, YONA adaptively aligns the
current frame's channel activation patterns with its adjacent reference frames
according to their foreground similarity. For the background, YONA conducts
background dynamic alignment guided by inter-frame difference to eliminate the
invalid features produced by drastic spatial jitters. Moreover, YONA applies
cross-frame contrastive learning during training, leveraging the ground truth
bounding box to improve the model's perception of polyp and background.
Quantitative and qualitative experiments on three public challenging benchmarks
demonstrate that our proposed YONA outperforms previous state-of-the-art
competitors by a large margin in both accuracy and speed.
