  An activation function is a crucial component of a neural network that
introduces non-linearity in the network. The state-of-the-art performance of a
neural network depends also on the perfect choice of an activation function. We
propose two novel non-monotonic smooth trainable activation functions, called
ErfAct and Pserf. Experiments suggest that the proposed functions improve the
network performance significantly compared to the widely used activations like
ReLU, Swish, and Mish. Replacing ReLU by ErfAct and Pserf, we have 5.68% and
5.42% improvement for top-1 accuracy on Shufflenet V2 (2.0x) network in
CIFAR100 dataset, 2.11% and 1.96% improvement for top-1 accuracy on Shufflenet
V2 (2.0x) network in CIFAR10 dataset, 1.0%, and 1.0% improvement on mean
average precision (mAP) on SSD300 model in Pascal VOC dataset.
