  In this paper, we study how the granularity of pretraining labels affects the
generalization of deep neural networks in image classification tasks. We focus
on the "fine-to-coarse" transfer learning setting, where the pretraining label
space is more fine-grained than that of the target problem. Empirically, we
show that pretraining on the leaf labels of ImageNet21k produces better
transfer results on ImageNet1k than pretraining on other coarser granularity
levels, which supports the common practice used in the community.
Theoretically, we explain the benefit of fine-grained pretraining by proving
that, for a data distribution satisfying certain hierarchy conditions, 1)
coarse-grained pretraining only allows a neural network to learn the "common"
or "easy-to-learn" features well, while 2) fine-grained pretraining helps the
network learn the "rarer" or "fine-grained" features in addition to the common
ones, thus improving its accuracy on hard downstream test samples in which
common features are missing or weak in strength. Furthermore, we perform
comprehensive experiments using the label hierarchies of iNaturalist 2021 and
observe that the following conditions, in addition to proper choice of label
granularity, enable the transfer to work well in practice: 1) the pretraining
dataset needs to have a meaningful label hierarchy, and 2) the pretraining and
target label functions need to align well.
