  Zero-shot learning methods typically assume that the new, unseen classes that
are encountered at deployment, come from the same distribution as training
classes. However, real-world scenarios often involve class distribution shifts
(e.g., in age or gender for person identification), posing challenges for
zero-shot classifiers that rely on learned representations from training
classes. In this work, we propose a model that assumes that the attribute
responsible for the shift is unknown in advance, and show that standard
training may lead to non-robust representations. To mitigate this, we propose
an algorithm for learning robust representations by (a) constructing synthetic
data environments via hierarchical sampling and (b) applying environment
balancing penalization, inspired by out-of-distribution problems. We show that
our approach improves generalization on diverse class distributions in both
simulations and real-world datasets.
