  We present Masked Frequency Modeling (MFM), a unified frequency-domain-based
approach for self-supervised pre-training of visual models. Instead of randomly
inserting mask tokens to the input embeddings in the spatial domain, in this
paper, we shift the perspective to the frequency domain. Specifically, MFM
first masks out a portion of frequency components of the input image and then
predicts the missing frequencies on the frequency spectrum. Our key insight is
that predicting masked components in the frequency domain is more ideal to
reveal underlying image patterns rather than predicting masked patches in the
spatial domain, due to the heavy spatial redundancy. Our findings suggest that
with the right configuration of mask-and-predict strategy, both the structural
information within high-frequency components and the low-level statistics among
low-frequency counterparts are useful in learning good representations. For the
first time, MFM demonstrates that, for both ViT and CNN, a simple non-Siamese
framework can learn meaningful representations even using none of the
following: (i) extra data, (ii) extra model, (iii) mask token. Experimental
results on image classification and semantic segmentation, as well as several
robustness benchmarks show the competitive performance and advanced robustness
of MFM compared with recent masked image modeling approaches. Furthermore, we
also comprehensively investigate the effectiveness of classical image
restoration tasks for representation learning from a unified frequency
perspective and reveal their intriguing relations with our MFM approach.
