  Nowadays artificial neural network models achieve remarkable results in many
disciplines. Functions mapping the representation provided by the model to the
probability distribution are the inseparable aspect of deep learning solutions.
Although softmax is a commonly accepted probability mapping function in the
machine learning community, it cannot return sparse outputs and always spreads
the positive probability to all positions. In this paper, we propose r-softmax,
a modification of the softmax, outputting sparse probability distribution with
controllable sparsity rate. In contrast to the existing sparse probability
mapping functions, we provide an intuitive mechanism for controlling the output
sparsity level. We show on several multi-label datasets that r-softmax
outperforms other sparse alternatives to softmax and is highly competitive with
the original softmax. We also apply r-softmax to the self-attention module of a
pre-trained transformer language model and demonstrate that it leads to
improved performance when fine-tuning the model on different natural language
processing tasks.
