  ImageNet-1K serves as the primary dataset for pretraining deep learning
models for computer vision tasks. ImageNet-21K dataset, which is bigger and
more diverse, is used less frequently for pretraining, mainly due to its
complexity, low accessibility, and underestimation of its added value. This
paper aims to close this gap, and make high-quality efficient pretraining on
ImageNet-21K available for everyone. Via a dedicated preprocessing stage,
utilization of WordNet hierarchical structure, and a novel training scheme
called semantic softmax, we show that various models significantly benefit from
ImageNet-21K pretraining on numerous datasets and tasks, including small
mobile-oriented models. We also show that we outperform previous ImageNet-21K
pretraining schemes for prominent new models like ViT and Mixer. Our proposed
pretraining pipeline is efficient, accessible, and leads to SoTA reproducible
results, from a publicly available dataset. The training code and pretrained
models are available at: https://github.com/Alibaba-MIIL/ImageNet21K
