  The hyper-parameters of a neural network are traditionally designed through a
time consuming process of trial and error that requires substantial expert
knowledge. Neural Architecture Search (NAS) algorithms aim to take the human
out of the loop by automatically finding a good set of hyper-parameters for the
problem at hand. These algorithms have mostly focused on hyper-parameters such
as the architectural configurations of the hidden layers and the connectivity
of the hidden neurons, but there has been relatively little work on automating
the search for completely new activation functions, which are one of the most
crucial hyper-parameters to choose. There are some widely used activation
functions nowadays which are simple and work well, but nonetheless, there has
been some interest in finding better activation functions. The work in the
literature has mostly focused on designing new activation functions by hand, or
choosing from a set of predefined functions while this work presents an
evolutionary algorithm to automate the search for completely new activation
functions. We compare these new evolved activation functions to other existing
and commonly used activation functions. The results are favorable and are
obtained from averaging the performance of the activation functions found over
30 runs, with experiments being conducted on 10 different datasets and
architectures to ensure the statistical robustness of the study.
