  This paper presents MOAT, a family of neural networks that build on top of
MObile convolution (i.e., inverted residual blocks) and ATtention. Unlike the
current works that stack separate mobile convolution and transformer blocks, we
effectively merge them into a MOAT block. Starting with a standard Transformer
block, we replace its multi-layer perceptron with a mobile convolution block,
and further reorder it before the self-attention operation. The mobile
convolution block not only enhances the network representation capacity, but
also produces better downsampled features. Our conceptually simple MOAT
networks are surprisingly effective, achieving 89.1% / 81.5% top-1 accuracy on
ImageNet-1K / ImageNet-1K-V2 with ImageNet22K pretraining. Additionally, MOAT
can be seamlessly applied to downstream tasks that require large resolution
inputs by simply converting the global attention to window attention. Thanks to
the mobile convolution that effectively exchanges local information between
pixels (and thus cross-windows), MOAT does not need the extra window-shifting
mechanism. As a result, on COCO object detection, MOAT achieves 59.2% box AP
with 227M model parameters (single-scale inference, and hard NMS), and on
ADE20K semantic segmentation, MOAT attains 57.6% mIoU with 496M model
parameters (single-scale inference). Finally, the tiny-MOAT family, obtained by
simply reducing the channel sizes, also surprisingly outperforms several
mobile-specific transformer-based models on ImageNet. The tiny-MOAT family is
also benchmarked on downstream tasks, serving as a baseline for the community.
We hope our simple yet effective MOAT will inspire more seamless integration of
convolution and self-attention. Code is publicly available.
