  In this paper, we conduct a study on the state-of-the-art methods for
text-to-image synthesis and propose a framework to evaluate these methods. We
consider syntheses where an image contains a single or multiple objects. Our
study outlines several issues in the current evaluation pipeline: (i) for image
quality assessment, a commonly used metric, e.g., Inception Score (IS), is
often either miscalibrated for the single-object case or misused for the
multi-object case; (ii) for text relevance and object accuracy assessment,
there is an overfitting phenomenon in the existing R-precision (RP) and
Semantic Object Accuracy (SOA) metrics, respectively; (iii) for multi-object
case, many vital factors for evaluation, e.g., object fidelity, positional
alignment, counting alignment, are largely dismissed; (iv) the ranking of the
methods based on current metrics is highly inconsistent with real images. To
overcome these issues, we propose a combined set of existing and new metrics to
systematically evaluate the methods. For existing metrics, we offer an improved
version of IS named IS* by using temperature scaling to calibrate the
confidence of the classifier used by IS; we also propose a solution to mitigate
the overfitting issues of RP and SOA. For new metrics, we develop counting
alignment, positional alignment, object-centric IS, and object-centric FID
metrics for evaluating the multi-object case. We show that benchmarking with
our bag of metrics results in a highly consistent ranking among existing
methods that is well-aligned with human evaluation. As a by-product, we create
AttnGAN++, a simple but strong baseline for the benchmark by stabilizing the
training of AttnGAN using spectral normalization. We also release our toolbox,
so-called TISE, for advocating fair and consistent evaluation of text-to-image
models.
