  A large body of recent work has identified transformations in the latent
spaces of generative adversarial networks (GANs) that consistently and
interpretably transform generated images. But existing techniques for
identifying these transformations rely on either a fixed vocabulary of
pre-specified visual concepts, or on unsupervised disentanglement techniques
whose alignment with human judgments about perceptual salience is unknown. This
paper introduces a new method for building open-ended vocabularies of primitive
visual concepts represented in a GAN's latent space. Our approach is built from
three components: (1) automatic identification of perceptually salient
directions based on their layer selectivity; (2) human annotation of these
directions with free-form, compositional natural language descriptions; and (3)
decomposition of these annotations into a visual concept vocabulary, consisting
of distilled directions labeled with single words. Experiments show that
concepts learned with our approach are reliable and composable -- generalizing
across classes, contexts, and observers, and enabling fine-grained manipulation
of image style and content.
