  A major contributor to the quality of a deep learning model is the selection
of the optimizer. We propose a new dual-joint search space in the realm of
neural optimizer search (NOS), along with an integrity check, to automate the
process of finding deep learning optimizers. Our dual-joint search space
simultaneously allows for the optimization of not only the update equation, but
also internal decay functions and learning rate schedules for optimizers. We
search the space using our proposed mutation-only, particle-based genetic
algorithm able to be massively parallelized for our domain-specific problem. We
evaluate our candidate optimizers on the CIFAR-10 dataset using a small
ConvNet. To assess generalization, the final optimizers were then transferred
to large-scale image classification on CIFAR- 100 and TinyImageNet, while also
being fine-tuned on Flowers102, Cars196, and Caltech101 using
EfficientNetV2Small. We found multiple optimizers, learning rate schedules, and
Adam variants that outperformed Adam, as well as other standard deep learning
optimizers, across the image classification tasks.
