  Motivated by the fact that forward and backward passes of a deep network
naturally form symmetric mappings between input and output representations, we
introduce a simple yet effective self-supervised vision model pretraining
framework inspired by energy-based models (EBMs). In the proposed framework, we
model energy estimation and data restoration as the forward and backward passes
of a single network without any auxiliary components, e.g., an extra decoder.
For the forward pass, we fit a network to an energy function that assigns low
energy scores to samples that belong to an unlabeled dataset, and high energy
otherwise. For the backward pass, we restore data from corrupted versions
iteratively using gradient-based optimization along the direction of energy
minimization. In this way, we naturally fold the encoder-decoder architecture
widely used in masked image modeling into the forward and backward passes of a
single vision model. Thus, our framework now accepts a wide range of pretext
tasks with different data corruption methods, and permits models to be
pretrained from masked image modeling, patch sorting, and image restoration,
including super-resolution, denoising, and colorization. We support our
findings with extensive experiments, and show the proposed method delivers
comparable and even better performance with remarkably fewer epochs of training
compared to the state-of-the-art self-supervised vision model pretraining
methods. Our findings shed light on further exploring self-supervised vision
model pretraining and pretext tasks beyond masked image modeling.
