  Inspired by the success of self-supervised autoregressive representation
learning in natural language (GPT and its variants), and advances in recent
visual architecture design with Vision Transformers (ViTs), in this paper, we
explore the effect various design choices have on the success of applying such
training strategies for visual feature learning. Specifically, we introduce a
novel strategy that we call Random Segments with Autoregressive Coding
(RandSAC). In RandSAC, we group patch representations (image tokens) into
hierarchically arranged segments; within each segment, tokens are predicted in
parallel, similar to BERT, while across segment predictions are sequential,
similar to GPT. We illustrate that randomized serialization of the segments
significantly improves the performance and results in distribution over
spatially-long (across-segments) and -short (within-segment) predictions which
are effective for feature learning. We illustrate the pertinence of these
design choices and explore alternatives on a number of datasets (e.g., CIFAR10,
CIFAR100, ImageNet). While our pre-training strategy works with a vanilla
Transformer, we also propose a conceptually simple, but highly effective,
addition to the decoder that allows learnable skip-connections to encoder$'$s
feature layers, which further improves the performance.
