  Concerns about interpretability, computational resources, and principled
inductive priors have motivated efforts to engineer sparse neural models for
NLP tasks. If sparsity is important for NLP, might well-trained neural models
naturally become roughly sparse? Using the Taxi-Euclidean norm to measure
sparsity, we find that frequent input words are associated with concentrated or
sparse activations, while frequent target words are associated with dispersed
activations but concentrated gradients. We find that gradients associated with
function words are more concentrated than the gradients of content words, even
controlling for word frequency.
