  This paper proposes a learnable nonlinear activation mechanism specifically
for convolutional neural network (CNN) termed as LENI, which learns to enhance
the negative information in CNNs. In sharp contrast to ReLU which cuts off the
negative neurons and suffers from the issue of ''dying ReLU'', LENI enjoys the
capacity to reconstruct the dead neurons and reduce the information loss.
Compared to improved ReLUs, LENI introduces a learnable approach to process the
negative phase information more properly. In this way, LENI can enhance the
model representational capacity significantly while maintaining the original
advantages of ReLU. As a generic activation mechanism, LENI possesses the
property of portability and can be easily utilized in any CNN models through
simply replacing the activation layers with LENI block. Extensive experiments
validate that LENI can improve the performance of various baseline models on
various benchmark datasets by a clear margin (up to 1.24% higher top-1 accuracy
on ImageNet-1k) with negligible extra parameters. Further experiments show that
LENI can act as a channel compensation mechanism, offering competitive or even
better performance but with fewer learned parameters than baseline models. In
addition, LENI introduces the asymmetry to the model structure which
contributes to the enhancement of representational capacity. Through
visualization experiments, we validate that LENI can retain more information
and learn more representations.
