  The accumulation of adaptations in an open-ended manner during lifetime
learning is a holy grail in reinforcement learning, intrinsic motivation,
artificial curiosity, and developmental robotics. We present a specification
for a cognitive architecture that is capable of specifying an unlimited range
of behaviors. We then give examples of how it can stochastically explore an
interesting space of adjacent possible behaviors. There are two main novelties;
the first is a proper definition of the fitness of self-generated games such
that interesting games are expected to evolve. The second is a modular and
evolvable behavior language that has systematicity, productivity, and
compositionality, i.e. it is a physical symbol system. A part of the
architecture has already been implemented on a humanoid robot.
