  Deep Neural Networks have shown great promise on a variety of downstream
applications; but their ability to adapt and generalize to new data and tasks
remains a challenge. However, the ability to perform few or zero-shot
adaptation to novel tasks is important for the scalability and deployment of
machine learning models. It is therefore crucial to understand what makes for
good, transfer-able features in deep networks that best allow for such
adaptation. In this paper, we shed light on this by showing that features that
are most transferable have high uniformity in the embedding space and propose a
uniformity regularization scheme that encourages better transfer and feature
reuse. We evaluate the regularization on its ability to facilitate adaptation
to unseen tasks and data, for which we conduct a thorough experimental study
covering four relevant, and distinct domains: few-shot Meta-Learning, Deep
Metric Learning, Zero-Shot Domain Adaptation, as well as Out-of-Distribution
classification. Across all experiments, we show that uniformity regularization
consistently offers benefits over baseline methods and is able to achieve
state-of-the-art performance in Deep Metric Learning and Meta-Learning.
