  With the widespread success of deep neural networks in science and
technology, it is becoming increasingly important to quantify the uncertainty
of the predictions produced by deep learning. In this paper, we introduce a new
method that attaches an explicit uncertainty statement to the probabilities of
classification using deep neural networks. Precisely, we view that the
classification probabilities are sampled from an unknown distribution, and we
propose to learn this distribution through the Dirichlet mixture that is
flexible enough for approximating any continuous distribution on the simplex.
We then construct credible intervals from the learned distribution to assess
the uncertainty of the classification probabilities. Our approach is easy to
implement, computationally efficient, and can be coupled with any deep neural
network architecture. Our method leverages the crucial observation that, in
many classification applications such as medical diagnosis, more than one class
labels are available for each observational unit. We demonstrate the usefulness
of our approach through simulations and a real data example.
