  We hypothesize that curiosity is a mechanism found by evolution that
encourages meaningful exploration early in an agent's life in order to expose
it to experiences that enable it to obtain high rewards over the course of its
lifetime. We formulate the problem of generating curious behavior as one of
meta-learning: an outer loop will search over a space of curiosity mechanisms
that dynamically adapt the agent's reward signal, and an inner loop will
perform standard reinforcement learning using the adapted reward signal.
However, current meta-RL methods based on transferring neural network weights
have only generalized between very similar tasks. To broaden the
generalization, we instead propose to meta-learn algorithms: pieces of code
similar to those designed by humans in ML papers. Our rich language of programs
combines neural networks with other building blocks such as buffers,
nearest-neighbor modules and custom loss functions. We demonstrate the
effectiveness of the approach empirically, finding two novel curiosity
algorithms that perform on par or better than human-designed published
curiosity algorithms in domains as disparate as grid navigation with image
inputs, acrobot, lunar lander, ant and hopper.
