  Universal Domain Adaptation (UniDA) deals with the problem of knowledge
transfer between two datasets with domain-shift as well as category-shift. The
goal is to categorize unlabeled target samples, either into one of the "known"
categories or into a single "unknown" category. A major problem in UniDA is
negative transfer, i.e. misalignment of "known" and "unknown" classes. To this
end, we first uncover an intriguing tradeoff between negative-transfer-risk and
domain-invariance exhibited at different layers of a deep network. It turns out
we can strike a balance between these two metrics at a mid-level layer. Towards
designing an effective framework based on this insight, we draw motivation from
Bag-of-visual-Words (BoW). Word-prototypes in a BoW-like representation of a
mid-level layer would represent lower-level visual primitives that are likely
to be unaffected by the category-shift in the high-level features. We develop
modifications that encourage learning of word-prototypes followed by
word-histogram based classification. Following this, subsidiary prototype-space
alignment (SPA) can be seen as a closed-set alignment problem, thereby avoiding
negative transfer. We realize this with a novel word-histogram-related pretext
task to enable closed-set SPA, operating in conjunction with goal task UniDA.
We demonstrate the efficacy of our approach on top of existing UniDA
techniques, yielding state-of-the-art performance across three standard UniDA
and Open-Set DA object recognition benchmarks.
