  Building robust deterministic neural networks remains a challenge. On the one
hand, some approaches improve out-of-distribution detection at the cost of
reducing classification accuracy in some situations. On the other hand, some
methods simultaneously increase classification accuracy, uncertainty
estimation, and out-of-distribution detection at the expense of reducing the
inference efficiency. In this paper, we propose training deterministic neural
networks using our DisMax loss, which works as a drop-in replacement for the
usual SoftMax loss (i.e., the combination of the linear output layer, the
SoftMax activation, and the cross-entropy loss). Starting from the IsoMax+
loss, we create each logit based on the distances to all prototypes, rather
than just the one associated with the correct class. We also introduce a
mechanism to combine images to construct what we call fractional probability
regularization. Moreover, we present a fast way to calibrate the network after
training. Finally, we propose a composite score to perform out-of-distribution
detection. Our experiments show that DisMax usually outperforms current
approaches simultaneously in classification accuracy, uncertainty estimation,
and out-of-distribution detection while maintaining deterministic neural
network inference efficiency. The code to reproduce the results is available at
https://github.com/dlmacedo/distinction-maximization-loss.
