  Semantic image synthesis aims to generate photo realistic images given a
semantic segmentation map. Despite much recent progress, training them still
requires large datasets of images annotated with per-pixel label maps that are
extremely tedious to obtain. To alleviate the high annotation cost, we propose
a transfer method that leverages a model trained on a large source dataset to
improve the learning ability on small target datasets via estimated pairwise
relations between source and target classes. The class affinity matrix is
introduced as a first layer to the source model to make it compatible with the
target label maps, and the source model is then further finetuned for the
target domain. To estimate the class affinities we consider different
approaches to leverage prior knowledge: semantic segmentation on the source
domain, textual label embeddings, and self-supervised vision features. We apply
our approach to GAN-based and diffusion-based architectures for semantic
synthesis. Our experiments show that the different ways to estimate class
affinity can be effectively combined, and that our approach significantly
improves over existing state-of-the-art transfer approaches for generative
image models.
