  One of the generally accepted views of modern deep learning is that
increasing the number of parameters usually leads to better quality. The two
easiest ways to increase the number of parameters is to increase the size of
the network, e.g. width, or to train a deep ensemble; both approaches improve
the performance in practice. In this work, we consider a fixed memory budget
setting, and investigate, what is more effective: to train a single wide
network, or to perform a memory split -- to train an ensemble of several
thinner networks, with the same total number of parameters? We find that, for
large enough budgets, the number of networks in the ensemble, corresponding to
the optimal memory split, is usually larger than one. Interestingly, this
effect holds for the commonly used sizes of the standard architectures. For
example, one WideResNet-28-10 achieves significantly worse test accuracy on
CIFAR-100 than an ensemble of sixteen thinner WideResNets: 80.6% and 82.52%
correspondingly. We call the described effect the Memory Split Advantage and
show that it holds for a variety of datasets and model architectures.
