  Ensembles over neural network weights trained from different random
initialization, known as deep ensembles, achieve state-of-the-art accuracy and
calibration. The recently introduced batch ensembles provide a drop-in
replacement that is more parameter efficient. In this paper, we design
ensembles not only over weights, but over hyperparameters to improve the state
of the art in both settings. For best performance independent of budget, we
propose hyper-deep ensembles, a simple procedure that involves a random search
over different hyperparameters, themselves stratified across multiple random
initializations. Its strong performance highlights the benefit of combining
models with both weight and hyperparameter diversity. We further propose a
parameter efficient version, hyper-batch ensembles, which builds on the layer
structure of batch ensembles and self-tuning networks. The computational and
memory costs of our method are notably lower than typical ensembles. On image
classification tasks, with MLP, LeNet, ResNet 20 and Wide ResNet 28-10
architectures, we improve upon both deep and batch ensembles.
