  The recent discovery of special human neocortical pyramidal neurons that can
individually learn the XOR function highlights the significant performance gap
between biological and artificial neurons. The output of these pyramidal
neurons first increases to a maximum with input and then decreases. Artificial
neurons with similar characteristics can be designed with oscillating
activation functions. Oscillating activation functions have multiple zeros
allowing single neurons to have multiple hyper-planes in their decision
boundary. This enables even single neurons to learn the XOR function. This
paper proposes four new oscillating activation functions inspired by human
pyramidal neurons that can also individually learn the XOR function.
Oscillating activation functions are non-saturating for all inputs unlike
popular activation functions, leading to improved gradient flow and faster
convergence. Using oscillating activation functions instead of popular
monotonic or non-monotonic single-zero activation functions enables neural
networks to train faster and solve classification problems with fewer layers.
An extensive comparison of 23 activation functions on CIFAR 10, CIFAR 100, and
Imagentte benchmarks is presented and the oscillating activation functions
proposed in this paper are shown to outperform all known popular activation
functions.
