  Recent research (arXiv:2310.11453, arXiv:2402.17764) has proposed binary and
ternary transformer networks as a way to significantly reduce memory and
improve inference speed in Large Language Models (LLMs) while maintaining
accuracy. In this work, we apply techniques from mechanistic interpretability
to investigate whether such networks learn distinctly different or similar
algorithms when compared to full-precision transformer networks. In particular,
we reverse engineer the algorithms learned for the toy problem of modular
addition where we find that binary and ternary networks learn similar
algorithms as full precision networks. This provides evidence against the
possibility of using binary and ternary networks as a more interpretable
alternative in the LLM setting.
