  We present a tree-structured network architecture for large scale image
classification. The trunk of the network contains convolutional layers
optimized over all classes. At a given depth, the trunk splits into separate
branches, each dedicated to discriminate a different subset of classes. Each
branch acts as an expert classifying a set of categories that are difficult to
tell apart, while the trunk provides common knowledge to all experts in the
form of shared features. The training of our "network of experts" is completely
end-to-end: the partition of categories into disjoint subsets is learned
simultaneously with the parameters of the network trunk and the experts are
trained jointly by minimizing a single learning objective over all classes. The
proposed structure can be built from any existing convolutional neural network
(CNN). We demonstrate its generality by adapting 4 popular CNNs for image
categorization into the form of networks of experts. Our experiments on
CIFAR100 and ImageNet show that in every case our method yields a substantial
improvement in accuracy over the base CNN, and gives the best result achieved
so far on CIFAR100. Finally, the improvement in accuracy comes at little
additional cost: compared to the base network, the training time is only
moderately increased and the number of parameters is comparable or in some
cases even lower.
