  Conventional model upgrades for visual search systems require offline refresh
of gallery features by feeding gallery images into new models (dubbed as
"backfill"), which is time-consuming and expensive, especially in large-scale
applications. The task of backward-compatible representation learning is
therefore introduced to support backfill-free model upgrades, where the new
query features are interoperable with the old gallery features. Despite the
success, previous works only investigated a close-set training scenario (i.e.,
the new training set shares the same classes as the old one), and are limited
by more realistic and challenging open-set scenarios. To this end, we first
introduce a new problem of universal backward-compatible representation
learning, covering all possible data split in model upgrades. We further
propose a simple yet effective method, dubbed as Universal Backward-Compatible
Training (UniBCT) with a novel structural prototype refinement algorithm, to
learn compatible representations in all kinds of model upgrading benchmarks in
a unified manner. Comprehensive experiments on the large-scale face recognition
datasets MS1Mv3 and IJB-C fully demonstrate the effectiveness of our method.
