  Despite the unresolved 'dying ReLU problem', the classical ReLU activation
function (AF) has been extensively applied in Deep Neural Networks (DNN), in
particular Convolutional Neural Networks (CNN), for image classification. The
common gradient issues of ReLU pose challenges in applications on academy and
industry sectors. Recent approaches for improvements are in a similar direction
by just proposing variations of the AF, such as Leaky ReLU (LReLU), while
maintaining the solution within the same unresolved gradient problems. In this
paper, the Absolute Leaky ReLU (ALReLU) AF, a variation of LReLU, is proposed,
as an alternative method to resolve the common 'dying ReLU problem' on NN-based
algorithms for supervised learning. The experimental results demonstrate that
by using the absolute values of LReLU's small negative gradient, has a
significant improvement in comparison with LReLU and ReLU, on image
classification of diseases such as COVID-19, text and tabular data
classification tasks on five different datasets.
