  Autoregressive language modeling (ALM) have been successfully used in
self-supervised pre-training in Natural language processing (NLP). However,
this paradigm has not achieved comparable results with other self-supervised
approach in computer vision (e.g., contrastive learning, mask image modeling).
In this paper, we try to find the reason why autoregressive modeling does not
work well on vision tasks. To tackle this problem, we fully analyze the
limitation of visual autoregressive methods and proposed a novel stochastic
autoregressive image modeling (named SAIM) by the two simple designs. First, we
employ stochastic permutation strategy to generate effective and robust image
context which is critical for vision tasks. Second, we create a parallel
encoder-decoder training process in which the encoder serves a similar role to
the standard vision transformer focus on learning the whole contextual
information, and meanwhile the decoder predicts the content of the current
position, so that the encoder and decoder can reinforce each other. By
introducing stochastic prediction and the parallel encoder-decoder, SAIM
significantly improve the performance of autoregressive image modeling. Our
method achieves the best accuracy (83.9%) on the vanilla ViT-Base model among
methods using only ImageNet-1K data. Transfer performance in downstream tasks
also show that our model achieves competitive performance.
