  Novel architectures have recently improved generative image synthesis leading
to excellent visual quality in various tasks. Much of this success is due to
the scalability of these architectures and hence caused by a dramatic increase
in model complexity and in the computational resources invested in training
these models. Our work questions the underlying paradigm of compressing large
training data into ever growing parametric representations. We rather present
an orthogonal, semi-parametric approach. We complement comparably small
diffusion or autoregressive models with a separate image database and a
retrieval strategy. During training we retrieve a set of nearest neighbors from
this external database for each training instance and condition the generative
model on these informative samples. While the retrieval approach is providing
the (local) content, the model is focusing on learning the composition of
scenes based on this content. As demonstrated by our experiments, simply
swapping the database for one with different contents transfers a trained model
post-hoc to a novel domain. The evaluation shows competitive performance on
tasks which the generative model has not been trained on, such as
class-conditional synthesis, zero-shot stylization or text-to-image synthesis
without requiring paired text-image data. With negligible memory and
computational overhead for the external database and retrieval we can
significantly reduce the parameter count of the generative model and still
outperform the state-of-the-art.
