  Diffusion models have become the de-facto approach for generating visual
data, which are trained to match the distribution of the training dataset. In
addition, we also want to control generation to fulfill desired properties such
as alignment to a text description, which can be specified with a black-box
reward function. Prior works fine-tune pretrained diffusion models to achieve
this goal through reinforcement learning-based algorithms. Nonetheless, they
suffer from issues including slow credit assignment as well as low quality in
their generated samples. In this work, we explore techniques that do not
directly maximize the reward but rather generate high-reward images with
relatively high probability -- a natural scenario for the framework of
generative flow networks (GFlowNets). To this end, we propose the Diffusion
Alignment with GFlowNet (DAG) algorithm to post-train diffusion models with
black-box property functions. Extensive experiments on Stable Diffusion and
various reward specifications corroborate that our method could effectively
align large-scale text-to-image diffusion models with given reward information.
