  Ensemble methods are arguably the most trustworthy techniques for boosting
the performance of machine learning models. Popular independent ensembles (IE)
relying on naive averaging/voting scheme have been of typical choice for most
applications involving deep neural networks, but they do not consider advanced
collaboration among ensemble models. In this paper, we propose new ensemble
methods specialized for deep neural networks, called confident multiple choice
learning (CMCL): it is a variant of multiple choice learning (MCL) via
addressing its overconfidence issue.In particular, the proposed major
components of CMCL beyond the original MCL scheme are (i) new loss, i.e.,
confident oracle loss, (ii) new architecture, i.e., feature sharing and (iii)
new training method, i.e., stochastic labeling. We demonstrate the effect of
CMCL via experiments on the image classification on CIFAR and SVHN, and the
foreground-background segmentation on the iCoseg. In particular, CMCL using 5
residual networks provides 14.05% and 6.60% relative reductions in the top-1
error rates from the corresponding IE scheme for the classification task on
CIFAR and SVHN, respectively.
