  Generative models are trained with the simple objective of imitating the
conditional probability distribution induced by the data they are trained on.
Therefore, when trained on data generated by humans, we may not expect the
artificial model to outperform the humans on their original objectives. In this
work, we study the phenomenon of transcendence: when a generative model
achieves capabilities that surpass the abilities of the experts generating its
data. We demonstrate transcendence by training an autoregressive transformer to
play chess from game transcripts, and show that the trained model can sometimes
achieve better performance than all players in the dataset. We theoretically
prove that transcendence can be enabled by low-temperature sampling, and
rigorously assess this claim experimentally. Finally, we discuss other sources
of transcendence, laying the groundwork for future investigation of this
phenomenon in a broader setting.
