  Combining multiple datasets enables performance boost on many computer vision
tasks. But similar trend has not been witnessed in object detection when
combining multiple datasets due to two inconsistencies among detection
datasets: taxonomy difference and domain gap. In this paper, we address these
challenges by a new design (named Detection Hub) that is dataset-aware and
category-aligned. It not only mitigates the dataset inconsistency but also
provides coherent guidance for the detector to learn across multiple datasets.
In particular, the dataset-aware design is achieved by learning a dataset
embedding that is used to adapt object queries as well as convolutional kernels
in detection heads. The categories across datasets are semantically aligned
into a unified space by replacing one-hot category representations with word
embedding and leveraging the semantic coherence of language embedding.
Detection Hub fulfills the benefits of large data on object detection.
Experiments demonstrate that joint training on multiple datasets achieves
significant performance gains over training on each dataset alone. Detection
Hub further achieves SoTA performance on UODB benchmark with wide variety of
datasets.
