  We consider the problem of uncertainty quantification in high dimensional
regression and classification for which deep ensemble have proven to be
promising methods. Recent observations have shown that deep ensemble often
return overconfident estimates outside the training domain, which is a major
limitation because shifted distributions are often encountered in real-life
scenarios. The principal challenge for this problem is to solve the trade-off
between increasing the diversity of the ensemble outputs and making accurate
in-distribution predictions. In this work, we show that an ensemble of networks
with large weights fitting the training data are likely to meet these two
objectives. We derive a simple and practical approach to produce such
ensembles, based on an original anti-regularization term penalizing small
weights and a control process of the weight increase which maintains the
in-distribution loss under an acceptable threshold. The developed approach does
not require any out-of-distribution training data neither any trade-off
hyper-parameter calibration. We derive a theoretical framework for this
approach and show that the proposed optimization can be seen as a
"water-filling" problem. Several experiments in both regression and
classification settings highlight that Deep Anti-Regularized Ensembles (DARE)
significantly improve uncertainty quantification outside the training domain in
comparison to recent deep ensembles and out-of-distribution detection methods.
All the conducted experiments are reproducible and the source code is available
at \url{https://github.com/antoinedemathelin/DARE}.
