  As generative AI becomes more prevalent, it is important to study how human
users interact with such models. In this work, we investigate how people use
text-to-image models to generate desired target images. To study this
interaction, we created ArtWhisperer, an online game where users are given a
target image and are tasked with iteratively finding a prompt that creates a
similar-looking image as the target. Through this game, we recorded over 50,000
human-AI interactions; each interaction corresponds to one text prompt created
by a user and the corresponding generated image. The majority of these are
repeated interactions where a user iterates to find the best prompt for their
target image, making this a unique sequential dataset for studying human-AI
collaborations. In an initial analysis of this dataset, we identify several
characteristics of prompt interactions and user strategies. People submit
diverse prompts and are able to discover a variety of text descriptions that
generate similar images. Interestingly, prompt diversity does not decrease as
users find better prompts. We further propose a new metric to quantify the
steerability of AI using our dataset. We define steerability as the expected
number of interactions required to adequately complete a task. We estimate this
value by fitting a Markov chain for each target task and calculating the
expected time to reach an adequate score in the Markov chain. We quantify and
compare AI steerability across different types of target images and two
different models, finding that images of cities and natural world images are
more steerable than artistic and fantasy images. These findings provide
insights into human-AI interaction behavior, present a concrete method of
assessing AI steerability, and demonstrate the general utility of the
ArtWhisperer dataset.
