  Reinforcement Learning from Human Feedback (RLHF) has shown potential in
qualitative tasks where easily defined performance measures are lacking.
However, there are drawbacks when RLHF is commonly used to optimize for average
human preferences, especially in generative tasks that demand diverse model
responses. Meanwhile, Quality Diversity (QD) algorithms excel at identifying
diverse and high-quality solutions but often rely on manually crafted diversity
metrics. This paper introduces Quality Diversity through Human Feedback (QDHF),
a novel approach that progressively infers diversity metrics from human
judgments of similarity among solutions, thereby enhancing the applicability
and effectiveness of QD algorithms in complex and open-ended domains. Empirical
studies show that QDHF significantly outperforms state-of-the-art methods in
automatic diversity discovery and matches the efficacy of QD with manually
crafted diversity metrics on standard benchmarks in robotics and reinforcement
learning. Notably, in open-ended generative tasks, QDHF substantially enhances
the diversity of text-to-image generation from a diffusion model and is more
favorably received in user studies. We conclude by analyzing QDHF's
scalability, robustness, and quality of derived diversity metrics, emphasizing
its strength in open-ended optimization tasks. Code and tutorials are available
at https://liding.info/qdhf.
