  Text-to-image generative models are capable of producing high-quality images
that often faithfully depict concepts described using natural language. In this
work, we comprehensively evaluate a range of text-to-image models on numerical
reasoning tasks of varying difficulty, and show that even the most advanced
models have only rudimentary numerical skills. Specifically, their ability to
correctly generate an exact number of objects in an image is limited to small
numbers, it is highly dependent on the context the number term appears in, and
it deteriorates quickly with each successive number. We also demonstrate that
models have poor understanding of linguistic quantifiers (such as "a few" or
"as many as"), the concept of zero, and struggle with more advanced concepts
such as partial quantities and fractional representations. We bundle prompts,
generated images and human annotations into GeckoNum, a novel benchmark for
evaluation of numerical reasoning.
