  We revisit the existing excellent Transformers from the perspective of
practical application. Most of them are not even as efficient as the basic
ResNets series and deviate from the realistic deployment scenario. It may be
due to the current criterion to measure computation efficiency, such as FLOPs
or parameters is one-sided, sub-optimal, and hardware-insensitive. Thus, this
paper directly treats the TensorRT latency on the specific hardware as an
efficiency metric, which provides more comprehensive feedback involving
computational capacity, memory cost, and bandwidth. Based on a series of
controlled experiments, this work derives four practical guidelines for
TensorRT-oriented and deployment-friendly network design, e.g., early CNN and
late Transformer at stage-level, early Transformer and late CNN at block-level.
Accordingly, a family of TensortRT-oriented Transformers is presented,
abbreviated as TRT-ViT. Extensive experiments demonstrate that TRT-ViT
significantly outperforms existing ConvNets and vision Transformers with
respect to the latency/accuracy trade-off across diverse visual tasks, e.g.,
image classification, object detection and semantic segmentation. For example,
at 82.7% ImageNet-1k top-1 accuracy, TRT-ViT is 2.7$\times$ faster than CSWin
and 2.0$\times$ faster than Twins. On the MS-COCO object detection task,
TRT-ViT achieves comparable performance with Twins, while the inference speed
is increased by 2.8$\times$.
