  Disentangling model activations into meaningful features is a central problem
in interpretability. However, the absence of ground-truth for these features in
realistic scenarios makes validating recent approaches, such as sparse
dictionary learning, elusive. To address this challenge, we propose a framework
for evaluating feature dictionaries in the context of specific tasks, by
comparing them against \emph{supervised} feature dictionaries. First, we
demonstrate that supervised dictionaries achieve excellent approximation,
control, and interpretability of model computations on the task. Second, we use
the supervised dictionaries to develop and contextualize evaluations of
unsupervised dictionaries along the same three axes.
  We apply this framework to the indirect object identification (IOI) task
using GPT-2 Small, with sparse autoencoders (SAEs) trained on either the IOI or
OpenWebText datasets. We find that these SAEs capture interpretable features
for the IOI task, but they are less successful than supervised features in
controlling the model. Finally, we observe two qualitative phenomena in SAE
training: feature occlusion (where a causally relevant concept is robustly
overshadowed by even slightly higher-magnitude ones in the learned features),
and feature over-splitting (where binary features split into many smaller, less
interpretable features). We hope that our framework will provide a useful step
towards more objective and grounded evaluations of sparse dictionary learning
methods.
