  Activation functions (AFs), which are pivotal to the success (or failure) of
a neural network, have received increased attention in recent years, with
researchers seeking to design novel AFs that improve some aspect of network
performance. In this paper we take another direction, wherein we combine a slew
of known AFs into successful architectures, proposing three methods to do so
beneficially: 1) generate AF architectures at random, 2) use Optuna, an
automatic hyper-parameter optimization software framework, with a
Tree-structured Parzen Estimator (TPE) sampler, and 3) use Optuna with a
Covariance Matrix Adaptation Evolution Strategy (CMA-ES) sampler. We show that
all methods often produce significantly better results for 25 classification
problems when compared with a standard network composed of ReLU hidden units
and a softmax output unit. Optuna with the TPE sampler emerged as the best AF
architecture-producing method.
