  Labelling of data for supervised learning can be costly and time-consuming
and the risk of incorporating label noise in large data sets is imminent. When
training a flexible discriminative model using a strictly proper loss, such
noise will inevitably shift the solution towards the conditional distribution
over noisy labels. Nevertheless, while deep neural networks have proven capable
of fitting random labels, regularisation and the use of robust loss functions
empirically mitigate the effects of label noise. However, such observations
concern robustness in accuracy, which is insufficient if reliable uncertainty
quantification is critical. We demonstrate this by analysing the properties of
the conditional distribution over noisy labels for an input-dependent noise
model. In addition, we evaluate the set of robust loss functions characterised
by noise-insensitive, asymptotic risk minimisers. We find that strictly proper
and robust loss functions both offer asymptotic robustness in accuracy, but
neither guarantee that the final model is calibrated. Moreover, even with
robust loss functions, overfitting is an issue in practice. With these results,
we aim to explain observed robustness of common training practices, such as
early stopping, to label noise. In addition, we aim to encourage the
development of new noise-robust algorithms that not only preserve accuracy but
that also ensure reliability.
