  Vision Transformer(ViT) is now dominating many vision tasks. The drawback of
quadratic complexity of its token-wise multi-head self-attention (MHSA), is
extensively addressed via either token sparsification or dimension reduction
(in spatial or channel). However, the therein redundancy of MHSA is usually
overlooked and so is the feed-forward network (FFN). To this end, we propose
attention map hallucination and FFN compaction to fill in the blank.
Specifically, we observe similar attention maps exist in vanilla ViT and
propose to hallucinate half of the attention maps from the rest with much
cheaper operations, which is called hallucinated-MHSA (hMHSA). As for FFN, we
factorize its hidden-to-output projection matrix and leverage the
re-parameterization technique to strengthen its capability, making it
compact-FFN (cFFN). With our proposed modules, a 10$\%$-20$\%$ reduction of
floating point operations (FLOPs) and parameters (Params) is achieved for
various ViT-based backbones, including straight (DeiT), hybrid (NextViT) and
hierarchical (PVT) structures, meanwhile, the performances are quite
competitive.
