  Deep neural networks have reached human-level performance on many computer
vision tasks. However, the objectives used to train these networks enforce only
that similar images are embedded at similar locations in the representation
space, and do not directly constrain the global structure of the resulting
space. Here, we explore the impact of supervising this global structure by
linearly aligning it with human similarity judgments. We find that a naive
approach leads to large changes in local representational structure that harm
downstream performance. Thus, we propose a novel method that aligns the global
structure of representations while preserving their local structure. This
global-local transform considerably improves accuracy across a variety of
few-shot learning and anomaly detection tasks. Our results indicate that human
visual representations are globally organized in a way that facilitates
learning from few examples, and incorporating this global structure into neural
network representations improves performance on downstream tasks.
