  System-oriented IR evaluations are limited to rather abstract understandings
of real user behavior. As a solution, simulating user interactions provides a
cost-efficient way to support system-oriented experiments with more realistic
directives when no interaction logs are available. While there are several user
models for simulated clicks or result list interactions, very few attempts have
been made towards query simulations, and it has not been investigated if these
can reproduce properties of real queries. In this work, we validate simulated
user query variants with the help of TREC test collections in reference to real
user queries that were made for the corresponding topics. Besides, we introduce
a simple yet effective method that gives better reproductions of real queries
than the established methods. Our evaluation framework validates the
simulations regarding the retrieval performance, reproducibility of topic score
distributions, shared task utility, effort and effect, and query term
similarity when compared with real user query variants. While the retrieval
effectiveness and statistical properties of the topic score distributions as
well as economic aspects are close to that of real queries, it is still
challenging to simulate exact term matches and later query reformulations.
