  Recently, DALL-E, a multimodal transformer language model, and its variants,
including diffusion models, have shown high-quality text-to-image generation
capabilities. However, despite the realistic image generation results, there
has not been a detailed analysis of how to evaluate such models. In this work,
we investigate the visual reasoning capabilities and social biases of different
text-to-image models, covering both multimodal transformer language models and
diffusion models. First, we measure three visual reasoning skills: object
recognition, object counting, and spatial relation understanding. For this, we
propose PaintSkills, a compositional diagnostic evaluation dataset that
measures these skills. Despite the high-fidelity image generation capability, a
large gap exists between the performance of recent models and the upper bound
accuracy in object counting and spatial relation understanding skills. Second,
we assess the gender and skin tone biases by measuring the gender/skin tone
distribution of generated images across various professions and attributes. We
demonstrate that recent text-to-image generation models learn specific biases
about gender and skin tone from web image-text pairs. We hope our work will
help guide future progress in improving text-to-image generation models on
visual reasoning skills and learning socially unbiased representations. Code
and data: https://github.com/j-min/DallEval
