  Given the recent impressive accomplishments of language models (LMs) for code
generation, we explore the use of LMs as adaptive mutation and crossover
operators for an evolutionary neural architecture search (NAS) algorithm. While
NAS still proves too difficult a task for LMs to succeed at solely through
prompting, we find that the combination of evolutionary prompt engineering with
soft prompt-tuning, a method we term EvoPrompting, consistently finds diverse
and high performing models. We first demonstrate that EvoPrompting is effective
on the computationally efficient MNIST-1D dataset, where EvoPrompting produces
convolutional architecture variants that outperform both those designed by
human experts and naive few-shot prompting in terms of accuracy and model size.
We then apply our method to searching for graph neural networks on the CLRS
Algorithmic Reasoning Benchmark, where EvoPrompting is able to design novel
architectures that outperform current state-of-the-art models on 21 out of 30
algorithmic reasoning tasks while maintaining similar model size. EvoPrompting
is successful at designing accurate and efficient neural network architectures
across a variety of machine learning tasks, while also being general enough for
easy adaptation to other tasks beyond neural network design.
