  Construction of a universal detector poses a crucial question: How can we
most effectively train a model on a large mixture of datasets? The answer lies
in learning dataset-specific features and ensembling their knowledge but do all
this in a single model. Previous methods achieve this by having separate
detection heads on a common backbone but that results in a significant increase
in parameters. In this work, we present Mixture-of-Experts as a solution,
highlighting that MoEs are much more than a scalability tool. We propose
Dataset-Aware Mixture-of-Experts, DAMEX where we train the experts to become an
`expert' of a dataset by learning to route each dataset tokens to its mapped
expert. Experiments on Universal Object-Detection Benchmark show that we
outperform the existing state-of-the-art by average +10.2 AP score and improve
over our non-MoE baseline by average +2.0 AP score. We also observe consistent
gains while mixing datasets with (1) limited availability, (2) disparate
domains and (3) divergent label sets. Further, we qualitatively show that DAMEX
is robust against expert representation collapse.
