  Cultural accumulation drives the open-ended and diverse progress in
capabilities spanning human history. It builds an expanding body of knowledge
and skills by combining individual exploration with inter-generational
information transmission. Despite its widespread success among humans, the
capacity for artificial learning agents to accumulate culture remains
under-explored. In particular, approaches to reinforcement learning typically
strive for improvements over only a single lifetime. Generational algorithms
that do exist fail to capture the open-ended, emergent nature of cultural
accumulation, which allows individuals to trade-off innovation and imitation.
Building on the previously demonstrated ability for reinforcement learning
agents to perform social learning, we find that training setups which balance
this with independent learning give rise to cultural accumulation. These
accumulating agents outperform those trained for a single lifetime with the
same cumulative experience. We explore this accumulation by constructing two
models under two distinct notions of a generation: episodic generations, in
which accumulation occurs via in-context learning and train-time generations,
in which accumulation occurs via in-weights learning. In-context and in-weights
cultural accumulation can be interpreted as analogous to knowledge and skill
accumulation, respectively. To the best of our knowledge, this work is the
first to present general models that achieve emergent cultural accumulation in
reinforcement learning, opening up new avenues towards more open-ended learning
systems, as well as presenting new opportunities for modelling human culture.
