  Meta Learning automates the search for learning algorithms. At the same time,
it creates a dependency on human engineering on the meta-level, where meta
learning algorithms need to be designed. In this paper, we investigate
self-referential meta learning systems that modify themselves without the need
for explicit meta optimization. We discuss the relationship of such systems to
in-context and memory-based meta learning and show that self-referential neural
networks require functionality to be reused in the form of parameter sharing.
Finally, we propose fitness monotonic execution (FME), a simple approach to
avoid explicit meta optimization. A neural network self-modifies to solve
bandit and classic control tasks, improves its self-modifications, and learns
how to learn, purely by assigning more computational resources to better
performing solutions.
