  Learning on big data brings success for artificial intelligence (AI), but the
annotation and training costs are expensive. In future, learning on small data
that approximates the generalization ability of big data is one of the ultimate
purposes of AI, which requires machines to recognize objectives and scenarios
relying on small data as humans. A series of learning topics is going on this
way such as active learning and few-shot learning. However, there are few
theoretical guarantees for their generalization performance. Moreover, most of
their settings are passive, that is, the label distribution is explicitly
controlled by finite training resources from known distributions. This survey
follows the agnostic active sampling theory under a PAC (Probably Approximately
Correct) framework to analyze the generalization error and label complexity of
learning on small data in model-agnostic supervised and unsupervised fashion.
Considering multiple learning communities could produce small data
representation and related topics have been well surveyed, we thus subjoin
novel geometric representation perspectives for small data: the Euclidean and
non-Euclidean (hyperbolic) mean, where the optimization solutions including the
Euclidean gradients, non-Euclidean gradients, and Stein gradient are presented
and discussed. Later, multiple learning communities that may be improved by
learning on small data are summarized, which yield data-efficient
representations, such as transfer learning, contrastive learning, graph
representation learning. Meanwhile, we find that the meta-learning may provide
effective parameter update policies for learning on small data. Then, we
explore multiple challenging scenarios for small data, such as the weak
supervision and multi-label. Finally, multiple data applications that may
benefit from efficient small data representation are surveyed.
