  Vision Transformers (ViTs) have recently dominated a range of computer vision
tasks, yet it suffers from low training data efficiency and inferior local
semantic representation capability without appropriate inductive bias.
Convolutional neural networks (CNNs) inherently capture regional-aware
semantics, inspiring researchers to introduce CNNs back into the architecture
of the ViTs to provide desirable inductive bias for ViTs. However, is the
locality achieved by the micro-level CNNs embedded in ViTs good enough? In this
paper, we investigate the problem by profoundly exploring how the macro
architecture of the hybrid CNNs/ViTs enhances the performances of hierarchical
ViTs. Particularly, we study the role of token embedding layers, alias
convolutional embedding (CE), and systemically reveal how CE injects desirable
inductive bias in ViTs. Besides, we apply the optimal CE configuration to 4
recently released state-of-the-art ViTs, effectively boosting the corresponding
performances. Finally, a family of efficient hybrid CNNs/ViTs, dubbed CETNets,
are released, which may serve as generic vision backbones. Specifically,
CETNets achieve 84.9% Top-1 accuracy on ImageNet-1K (training from scratch),
48.6% box mAP on the COCO benchmark, and 51.6% mIoU on the ADE20K,
substantially improving the performances of the corresponding state-of-the-art
baselines.
