  Text-to-image generation has recently emerged as a viable alternative to
text-to-image retrieval, due to the visually impressive results of generative
diffusion models. Although query performance prediction is an active research
topic in information retrieval, to the best of our knowledge, there is no prior
study that analyzes the difficulty of queries (prompts) in text-to-image
generation, based on human judgments. To this end, we introduce the first
dataset of prompts which are manually annotated in terms of image generation
performance. In order to determine the difficulty of the same prompts in image
retrieval, we also collect manual annotations that represent retrieval
performance. We thus propose the first benchmark for joint text-to-image prompt
and query performance prediction, comprising 10K queries. Our benchmark
enables: (i) the comparative assessment of the difficulty of prompts/queries in
image generation and image retrieval, and (ii) the evaluation of prompt/query
performance predictors addressing both generation and retrieval. We present
results with several pre-generation/retrieval and post-generation/retrieval
performance predictors, thus providing competitive baselines for future
research. Our benchmark and code is publicly available under the CC BY 4.0
license at https://github.com/Eduard6421/PQPP.
