  The colorectal polyps classification is a critical clinical examination. To
improve the classification accuracy, most computer-aided diagnosis algorithms
recognize colorectal polyps by adopting Narrow-Band Imaging (NBI). However, the
NBI usually suffers from missing utilization in real clinic scenarios since the
acquisition of this specific image requires manual switching of the light mode
when polyps have been detected by using White-Light (WL) images. To avoid the
above situation, we propose a novel method to directly achieve accurate
white-light colonoscopy image classification by conducting structured
cross-modal representation consistency. In practice, a pair of multi-modal
images, i.e. NBI and WL, are fed into a shared Transformer to extract
hierarchical feature representations. Then a novel designed Spatial Attention
Module (SAM) is adopted to calculate the similarities between the class token
and patch tokens %from multi-levels for a specific modality image. By aligning
the class tokens and spatial attention maps of paired NBI and WL images at
different levels, the Transformer achieves the ability to keep both global and
local representation consistency for the above two modalities. Extensive
experimental results illustrate the proposed method outperforms the recent
studies with a margin, realizing multi-modal prediction with a single
Transformer while greatly improving the classification accuracy when only with
WL images.
