  The performance of artificial neural networks (ANNs) is influenced by weight
initialization, the nature of activation functions, and their architecture.
There is a wide range of activation functions that are traditionally used to
train a neural network, e.g. sigmoid, tanh, and Rectified Linear Unit (ReLU). A
widespread practice is to use the same type of activation function in all
neurons in a given layer. In this manuscript, we present a type of neural
network in which the activation functions in every layer form a polynomial
basis; we name this method SWAG after the initials of the last names of the
authors. We tested SWAG on three complex highly non-linear functions as well as
the MNIST handwriting data set. SWAG outperforms and converges faster than the
state of the art performance in fully connected neural networks. Given the low
computational complexity of SWAG, and the fact that it was capable of solving
problems current architectures cannot, it has the potential to change the way
that we approach deep learning.
