  We introduce an approach to bias deep generative models, such as GANs and
diffusion models, towards generating data with either enhanced fidelity or
increased diversity. Our approach involves manipulating the distribution of
training and generated data through a novel metric for individual samples,
named pseudo density, which is based on the nearest-neighbor information from
real samples. Our approach offers three distinct techniques to adjust the
fidelity and diversity of deep generative models: 1) Per-sample perturbation,
enabling precise adjustments for individual samples towards either more common
or more unique characteristics; 2) Importance sampling during model inference
to enhance either fidelity or diversity in the generated data; 3) Fine-tuning
with importance sampling, which guides the generative model to learn an
adjusted distribution, thus controlling fidelity and diversity. Furthermore,
our fine-tuning method demonstrates the ability to improve the Frechet
Inception Distance (FID) for pre-trained generative models with minimal
iterations.
