  Domain adaptation for semantic segmentation across datasets consisting of the
same categories has seen several recent successes. However, a more general
scenario is when the source and target datasets correspond to non-overlapping
label spaces. For example, categories in segmentation datasets change vastly
depending on the type of environment or application, yet share many valuable
semantic relations. Existing approaches based on feature alignment or
discrepancy minimization do not take such category shift into account. In this
work, we present Cluster-to-Adapt (C2A), a computationally efficient
clustering-based approach for domain adaptation across segmentation datasets
with completely different, but possibly related categories. We show that such a
clustering objective enforced in a transformed feature space serves to
automatically select categories across source and target domains that can be
aligned for improving the target performance, while preventing negative
transfer for unrelated categories. We demonstrate the effectiveness of our
approach through experiments on the challenging problem of outdoor to indoor
adaptation for semantic segmentation in few-shot as well as zero-shot settings,
with consistent improvements in performance over existing approaches and
baselines in all cases.
