  Despite impressive progress in deep learning, generalizing far beyond the
training distribution is an important open challenge. In this work, we consider
few-shot classification, and aim to shed light on what makes some novel classes
easier to learn than others, and what types of learned representations
generalize better. To this end, we define a new paradigm in terms of attributes
-- simple building blocks of which concepts are formed -- as a means of
quantifying the degree of relatedness of different concepts. Our empirical
analysis reveals that supervised learning generalizes poorly to new attributes,
but a combination of self-supervised pretraining with supervised finetuning
leads to stronger generalization. The benefit of self-supervised pretraining
and supervised finetuning is further investigated through controlled
experiments using random splits of the attribute space, and we find that
predictability of test attributes provides an informative estimate of a model's
generalization ability.
