  Generalized Category Discovery (GCD) is a challenging task in which, given a
partially labelled dataset, models must categorize all unlabelled instances,
regardless of whether they come from labelled categories or from new ones. In
this paper, we challenge a remaining assumption in this task: that all images
share the same domain. Specifically, we introduce a new task and method to
handle GCD when the unlabelled data also contains images from different domains
to the labelled set. Our proposed `HiLo' networks extract High-level semantic
and Low-level domain features, before minimizing the mutual information between
the representations. Our intuition is that the clusterings based on domain
information and semantic information should be independent. We further extend
our method with a specialized domain augmentation tailored for the GCD task, as
well as a curriculum learning approach. Finally, we construct a benchmark from
corrupted fine-grained datasets as well as a large-scale evaluation on
DomainNet with real-world domain shifts, reimplementing a number of GCD
baselines in this setting. We demonstrate that HiLo outperforms SoTA category
discovery models by a large margin on all evaluations.
