  We here propose a novel hierarchical transformer model that adeptly
integrates the feature extraction capabilities of Convolutional Neural Networks
(CNNs) with the advanced representational potential of Vision Transformers
(ViTs). Addressing the lack of inductive biases and dependence on extensive
training datasets in ViTs, our model employs a CNN backbone to generate
hierarchical visual representations. These representations are then adapted for
transformer input through an innovative patch tokenization. We also introduce a
'scale attention' mechanism that captures cross-scale dependencies,
complementing patch attention to enhance spatial understanding and preserve
global perception. Our approach significantly outperforms baseline models on
small and medium-sized medical datasets, demonstrating its efficiency and
generalizability. The components are designed as plug-and-play for different
CNN architectures and can be adapted for multiple applications. The code is
available at https://github.com/xiaoyatang/DuoFormer.git.
