  Motivated by biological evolution, this paper explains the rationality of
Vision Transformer by analogy with the proven practical Evolutionary Algorithm
(EA) and derives that both have consistent mathematical formulation. Then
inspired by effective EA variants, we propose a novel pyramid EATFormer
backbone that only contains the proposed \emph{EA-based Transformer} (EAT)
block, which consists of three residual parts, i.e., \emph{Multi-Scale Region
Aggregation} (MSRA), \emph{Global and Local Interaction} (GLI), and
\emph{Feed-Forward Network} (FFN) modules, to model multi-scale, interactive,
and individual information separately. Moreover, we design a \emph{Task-Related
Head} (TRH) docked with transformer backbone to complete final information
fusion more flexibly and \emph{improve} a \emph{Modulated Deformable MSA}
(MD-MSA) to dynamically model irregular locations. Massive quantitative and
quantitative experiments on image classification, downstream tasks, and
explanatory experiments demonstrate the effectiveness and superiority of our
approach over State-Of-The-Art (SOTA) methods. \Eg, our Mobile (1.8M), Tiny
(6.1M), Small (24.3M), and Base (49.0M) models achieve 69.4, 78.4, 83.1, and
83.9 Top-1 only trained on ImageNet-1K with naive training recipe;
EATFormer-Tiny/Small/Base armed Mask-R-CNN obtain 45.4/47.4/49.0 box AP and
41.4/42.9/44.2 mask AP on COCO detection, surpassing contemporary MPViT-T,
Swin-T, and Swin-S by 0.6/1.4/0.5 box AP and 0.4/1.3/0.9 mask AP separately
with less FLOPs; Our EATFormer-Small/Base achieve 47.3/49.3 mIoU on ADE20K by
Upernet that exceeds Swin-T/S by 2.8/1.7. Code is available at
\url{https://github.com/zhangzjn/EATFormer}.
