  Universal image representations are critical in enabling real-world
fine-grained and instance-level recognition applications, where objects and
entities from any domain must be identified at large scale. Despite recent
advances, existing methods fail to capture important domain-specific knowledge,
while also ignoring differences in data distribution across different domains.
This leads to a large performance gap between efficient universal solutions and
expensive approaches utilising a collection of specialist models, one for each
domain. In this work, we make significant strides towards closing this gap, by
introducing a new learning technique, dubbed UDON (Universal Dynamic Online
DistillatioN). UDON employs multi-teacher distillation, where each teacher is
specialized in one domain, to transfer detailed domain-specific knowledge into
the student universal embedding. UDON's distillation approach is not only
effective, but also very efficient, by sharing most model parameters between
the student and all teachers, where all models are jointly trained in an online
manner. UDON also comprises a sampling technique which adapts the training
process to dynamically allocate batches to domains which are learned slower and
require more frequent processing. This boosts significantly the learning of
complex domains which are characterised by a large number of classes and
long-tail distributions. With comprehensive experiments, we validate each
component of UDON, and showcase significant improvements over the state of the
art in the recent UnED benchmark. Code: https://github.com/nikosips/UDON .
