  Some neurons in deep networks specialize in recognizing highly specific
perceptual, structural, or semantic features of inputs. In computer vision,
techniques exist for identifying neurons that respond to individual concept
categories like colors, textures, and object classes. But these techniques are
limited in scope, labeling only a small subset of neurons and behaviors in any
network. Is a richer characterization of neuron-level computation possible? We
introduce a procedure (called MILAN, for mutual-information-guided linguistic
annotation of neurons) that automatically labels neurons with open-ended,
compositional, natural language descriptions. Given a neuron, MILAN generates a
description by searching for a natural language string that maximizes pointwise
mutual information with the image regions in which the neuron is active. MILAN
produces fine-grained descriptions that capture categorical, relational, and
logical structure in learned features. These descriptions obtain high agreement
with human-generated feature descriptions across a diverse set of model
architectures and tasks, and can aid in understanding and controlling learned
models. We highlight three applications of natural language neuron
descriptions. First, we use MILAN for analysis, characterizing the distribution
and importance of neurons selective for attribute, category, and relational
information in vision models. Second, we use MILAN for auditing, surfacing
neurons sensitive to human faces in datasets designed to obscure them. Finally,
we use MILAN for editing, improving robustness in an image classifier by
deleting neurons sensitive to text features spuriously correlated with class
labels.
