  Activation functions play a vital role in the training of Convolutional
Neural Networks. For this reason, to develop efficient and performing functions
is a crucial problem in the deep learning community. Key to these approaches is
to permit a reliable parameter learning, avoiding vanishing gradient problems.
The goal of this work is to propose an ensemble of Convolutional Neural
Networks trained using several different activation functions. Moreover, a
novel activation function is here proposed for the first time. Our aim is to
improve the performance of Convolutional Neural Networks in small/medium size
biomedical datasets. Our results clearly show that the proposed ensemble
outperforms Convolutional Neural Networks trained with standard ReLU as
activation function. The proposed ensemble outperforms with a p-value of 0.01
each tested stand-alone activation function; for reliable performance
comparison we have tested our approach in more than 10 datasets, using two
well-known Convolutional Neural Network: Vgg16 and ResNet50. MATLAB code used
here will be available at https://github.com/LorisNanni.
