  Supervised learning typically focuses on learning transferable
representations from training examples annotated by humans. While rich
annotations (like soft labels) carry more information than sparse annotations
(like hard labels), they are also more expensive to collect. For example, while
hard labels only provide information about the closest class an object belongs
to (e.g., "this is a dog"), soft labels provide information about the object's
relationship with multiple classes (e.g., "this is most likely a dog, but it
could also be a wolf or a coyote"). We use information theory to compare how a
number of commonly-used supervision signals contribute to
representation-learning performance, as well as how their capacity is affected
by factors such as the number of labels, classes, dimensions, and noise. Our
framework provides theoretical justification for using hard labels in the
big-data regime, but richer supervision signals for few-shot learning and
out-of-distribution generalization. We validate these results empirically in a
series of experiments with over 1 million crowdsourced image annotations and
conduct a cost-benefit analysis to establish a tradeoff curve that enables
users to optimize the cost of supervising representation learning on their own
datasets.
