  This paper introduces an efficient and robust method for discovering
interpretable circuits in large language models using discrete sparse
autoencoders. Our approach addresses key limitations of existing techniques,
namely computational complexity and sensitivity to hyperparameters. We propose
training sparse autoencoders on carefully designed positive and negative
examples, where the model can only correctly predict the next token for the
positive examples. We hypothesise that learned representations of attention
head outputs will signal when a head is engaged in specific computations. By
discretising the learned representations into integer codes and measuring the
overlap between codes unique to positive examples for each head, we enable
direct identification of attention heads involved in circuits without the need
for expensive ablations or architectural modifications. On three well-studied
tasks - indirect object identification, greater-than comparisons, and docstring
completion - the proposed method achieves higher precision and recall in
recovering ground-truth circuits compared to state-of-the-art baselines, while
reducing runtime from hours to seconds. Notably, we require only 5-10 text
examples for each task to learn robust representations. Our findings highlight
the promise of discrete sparse autoencoders for scalable and efficient
mechanistic interpretability, offering a new direction for analysing the inner
workings of large language models.
