  Universality is a key hypothesis in mechanistic interpretability -- that
different models learn similar features and circuits when trained on similar
tasks. In this work, we study the universality hypothesis by examining how
small neural networks learn to implement group composition. We present a novel
algorithm by which neural networks may implement composition for any finite
group via mathematical representation theory. We then show that networks
consistently learn this algorithm by reverse engineering model logits and
weights, and confirm our understanding using ablations. By studying networks of
differing architectures trained on various groups, we find mixed evidence for
universality: using our algorithm, we can completely characterize the family of
circuits and features that networks learn on this task, but for a given network
the precise circuits learned -- as well as the order they develop -- are
arbitrary.
