  Deep learning models for vision tasks are trained on large datasets under the
assumption that there exists a universal representation that can be used to
make predictions for all samples. Whereas high complexity models are proven to
be capable of learning such representations, a mixture of experts trained on
specific subsets of the data can infer the labels more efficiently. However
using mixture of experts poses two new problems, namely (i) assigning the
correct expert at inference time when a new unseen sample is presented. (ii)
Finding the optimal partitioning of the training data, such that the experts
rely the least on common features. In Dynamic Routing (DR) a novel architecture
is proposed where each layer is composed of a set of experts, however without
addressing the two challenges we demonstrate that the model reverts to using
the same subset of experts.
  In our method, Diversified Dynamic Routing (DivDR) the model is explicitly
trained to solve the challenge of finding relevant partitioning of the data and
assigning the correct experts in an unsupervised approach. We conduct several
experiments on semantic segmentation on Cityscapes and object detection and
instance segmentation on MS-COCO showing improved performance over several
baselines.
