  This paper presents a new perspective of self-supervised learning based on
extending heat equation into high dimensional feature space. In particular, we
remove time dependence by steady-state condition, and extend the remaining 2D
Laplacian from x--y isotropic to linear correlated. Furthermore, we simplify it
by splitting x and y axes as two first-order linear differential equations.
Such simplification explicitly models the spatial invariance along horizontal
and vertical directions separately, supporting prediction across image blocks.
This introduces a very simple masked image modeling (MIM) method, named
QB-Heat.
  QB-Heat leaves a single block with size of quarter image unmasked and
extrapolates other three masked quarters linearly. It brings MIM to CNNs
without bells and whistles, and even works well for pre-training light-weight
networks that are suitable for both image classification and object detection
without fine-tuning. Compared with MoCo-v2 on pre-training a Mobile-Former with
5.8M parameters and 285M FLOPs, QB-Heat is on par in linear probing on
ImageNet, but clearly outperforms in non-linear probing that adds a transformer
block before linear classifier (65.6% vs. 52.9%). When transferring to object
detection with frozen backbone, QB-Heat outperforms MoCo-v2 and supervised
pre-training on ImageNet by 7.9 and 4.5 AP respectively.
  This work provides an insightful hypothesis on the invariance within visual
representation over different shapes and textures: the linear relationship
between horizontal and vertical derivatives. The code will be publicly
released.
