  We propose integrally pre-trained transformer pyramid network (iTPN), towards
jointly optimizing the network backbone and the neck, so that transfer gap
between representation models and downstream tasks is minimal. iTPN is born
with two elaborated designs: 1) The first pre-trained feature pyramid upon
vision transformer (ViT). 2) Multi-stage supervision to the feature pyramid
using masked feature modeling (MFM). iTPN is updated to Fast-iTPN, reducing
computational memory overhead and accelerating inference through two flexible
designs. 1) Token migration: dropping redundant tokens of the backbone while
replenishing them in the feature pyramid without attention operations. 2) Token
gathering: reducing computation cost caused by global attention by introducing
few gathering tokens. The base/large-level Fast-iTPN achieve 88.75%/89.5% top-1
accuracy on ImageNet-1K. With 1x training schedule using DINO, the
base/large-level Fast-iTPN achieves 58.4%/58.8% box AP on COCO object
detection, and a 57.5%/58.7% mIoU on ADE20K semantic segmentation using
MaskDINO. Fast-iTPN can accelerate the inference procedure by up to 70%, with
negligible performance loss, demonstrating the potential to be a powerful
backbone for downstream vision tasks. The code is available at:
github.com/sunsmarterjie/iTPN.
