  The application of the deep learning model in classification plays an
important role in the accurate detection of the target objects. However, the
accuracy is affected by the activation function in the hidden and output layer.
In this paper, an activation function called TaLU, which is a combination of
Tanh and Rectified Linear Units (ReLU), is used to improve the prediction. ReLU
activation function is used by many deep learning researchers for its
computational efficiency, ease of implementation, intuitive nature, etc.
However, it suffers from a dying gradient problem. For instance, when the input
is negative, its output is always zero because its gradient is zero. A number
of researchers used different approaches to solve this issue. Some of the most
notable are LeakyReLU, Softplus, Softsign, ELU, ThresholdedReLU, etc. This
research developed TaLU, a modified activation function combining Tanh and
ReLU, which mitigates the dying gradient problem of ReLU. The deep learning
model with the proposed activation function was tested on MNIST and CIFAR-10,
and it outperforms ReLU and some other studied activation functions in terms of
accuracy(upto 6% in most cases, when used with Batch Normalization and a
reasonable learning rate).
