  Determining the necessity of resecting malignant polyps during colonoscopy
screen is crucial for patient outcomes, yet challenging due to the
time-consuming and costly nature of histopathology examination. While deep
learning-based classification models have shown promise in achieving optical
biopsy with endoscopic images, they often suffer from a lack of explainability.
To overcome this limitation, we introduce EndoFinder, a content-based image
retrieval framework to find the 'digital twin' polyp in the reference database
given a newly detected polyp. The clinical semantics of the new polyp can be
inferred referring to the matched ones. EndoFinder pioneers a polyp-aware image
encoder that is pre-trained on a large polyp dataset in a self-supervised way,
merging masked image modeling with contrastive learning. This results in a
generic embedding space ready for different downstream clinical tasks based on
image retrieval. We validate the framework on polyp re-identification and
optical biopsy tasks, with extensive experiments demonstrating that EndoFinder
not only achieves explainable diagnostics but also matches the performance of
supervised classification models. EndoFinder's reliance on image retrieval has
the potential to support diverse downstream decision-making tasks during
real-time colonoscopy procedures.
