  Transformer is a potentially powerful architecture for vision tasks. Although
equipped with more parameters and attention mechanism, its performance is not
as dominant as CNN currently. CNN is usually computationally cheaper and still
the leading competitor in various vision tasks. One research direction is to
adopt the successful ideas of CNN and improve transformer, but it often relies
on elaborated and heuristic network design. Observing that transformer and CNN
are complementary in representation learning and convergence speed, we propose
an efficient training framework called Vision Pair Learning (VPL) for image
classification task. VPL builds up a network composed of a transformer branch,
a CNN branch and pair learning module. With multi-stage training strategy, VPL
enables the branches to learn from their partners during the appropriate stage
of the training process, and makes them both achieve better performance with
less time cost. Without external data, VPL promotes the top-1 accuracy of
ViT-Base and ResNet-50 on the ImageNet-1k validation set to 83.47% and 79.61%
respectively. Experiments on other datasets of various domains prove the
efficacy of VPL and suggest that transformer performs better when paired with
the differently structured CNN in VPL. we also analyze the importance of
components through ablation study.
