  Deep neural networks have become the default choice for many of the machine
learning tasks such as classification and regression. Dropout, a method
commonly used to improve the convergence of deep neural networks, generates an
ensemble of thinned networks with extensive weight sharing. Recent studies that
dropout can be viewed as an approximate variational inference in Gaussian
processes, and used as a practical tool to obtain uncertainty estimates of the
network. We propose a novel statistical mechanics based framework to dropout
and use this framework to propose a new generic algorithm that focuses on
estimates of the variance of the loss as measured by the ensemble of thinned
networks. Our approach can be applied to a wide range of deep neural network
architectures and machine learning tasks. In classification, this algorithm
allows the generation of a don't-know answer to be generated, which can
increase the reliability of the classifier. Empirically we demonstrate
state-of-the-art AUC results on publicly available benchmarks.
