  Despite rapid adoption and deployment of large language models (LLMs), the
internal computations of these models remain opaque and poorly understood. In
this work, we seek to understand how high-level human-interpretable features
are represented within the internal neuron activations of LLMs. We train
$k$-sparse linear classifiers (probes) on these internal activations to predict
the presence of features in the input; by varying the value of $k$ we study the
sparsity of learned representations and how this varies with model scale. With
$k=1$, we localize individual neurons which are highly relevant for a
particular feature, and perform a number of case studies to illustrate general
properties of LLMs. In particular, we show that early layers make use of sparse
combinations of neurons to represent many features in superposition, that
middle layers have seemingly dedicated neurons to represent higher-level
contextual features, and that increasing scale causes representational sparsity
to increase on average, but there are multiple types of scaling dynamics. In
all, we probe for over 100 unique features comprising 10 different categories
in 7 different models spanning 70 million to 6.9 billion parameters.
