  Over the past few years, softmax and SGD have become a commonly used
component and the default training strategy in CNN frameworks, respectively.
However, when optimizing CNNs with SGD, the saturation behavior behind softmax
always gives us an illusion of training well and then is omitted. In this
paper, we first emphasize that the early saturation behavior of softmax will
impede the exploration of SGD, which sometimes is a reason for model converging
at a bad local-minima, then propose Noisy Softmax to mitigating this early
saturation issue by injecting annealed noise in softmax during each iteration.
This operation based on noise injection aims at postponing the early saturation
and further bringing continuous gradients propagation so as to significantly
encourage SGD solver to be more exploratory and help to find a better
local-minima. This paper empirically verifies the superiority of the early
softmax desaturation, and our method indeed improves the generalization ability
of CNN model by regularization. We experimentally find that this early
desaturation helps optimization in many tasks, yielding state-of-the-art or
competitive results on several popular benchmark datasets.
