  Recent advancements in Deep Neural Network (DNN) models have significantly
improved performance across computer vision tasks. However, achieving highly
generalizable and high-performing vision models requires expansive datasets,
resulting in significant storage requirements. This storage challenge is a
critical bottleneck for scaling up models. A recent breakthrough by SeiT
proposed the use of Vector-Quantized (VQ) feature vectors (i.e., tokens) as
network inputs for vision classification. This approach achieved 90% of the
performance of a model trained on full-pixel images with only 1% of the
storage. While SeiT needs labeled data, its potential in scenarios beyond fully
supervised learning remains largely untapped. In this paper, we extend SeiT by
integrating Masked Token Modeling (MTM) for self-supervised pre-training.
Recognizing that self-supervised approaches often demand more data due to the
lack of labels, we introduce TokenAdapt and ColorAdapt. These methods
facilitate comprehensive token-friendly data augmentation, effectively
addressing the increased data requirements of self-supervised learning. We
evaluate our approach across various scenarios, including storage-efficient
ImageNet-1k classification, fine-grained classification, ADE-20k semantic
segmentation, and robustness benchmarks. Experimental results demonstrate
consistent performance improvement in diverse experiments, validating the
effectiveness of our method. Code is available at
https://github.com/naver-ai/seit.
