  Understanding the internal representations of large language models (LLMs)
can help explain models' behavior and verify their alignment with human values.
Given the capabilities of LLMs in generating human-understandable text, we
propose leveraging the model itself to explain its internal representations in
natural language. We introduce a framework called Patchscopes and show how it
can be used to answer a wide range of questions about an LLM's computation. We
show that many prior interpretability methods based on projecting
representations into the vocabulary space and intervening on the LLM
computation can be viewed as instances of this framework. Moreover, several of
their shortcomings such as failure in inspecting early layers or lack of
expressivity can be mitigated by Patchscopes. Beyond unifying prior inspection
techniques, Patchscopes also opens up new possibilities such as using a more
capable model to explain the representations of a smaller model, and multihop
reasoning error correction.
