  Treasure Data is processing millions of distributed SQL queries every day on
the cloud. Upgrading the query engine service at this scale is challenging
because we need to migrate all of the production queries of the customers to a
new version while preserving the correctness and performance of the data
processing pipelines. To ensure the quality of the query engines, we utilize
our query logs to build customer-specific benchmarks and replay these queries
with real customer data in a secure pre-production environment. To simulate
millions of queries, we need effective minimization of test query sets and
better reporting of the simulation results to proactively find incompatible
changes and performance regression of the new version. This paper describes the
overall design of our system and shares various challenges in maintaining the
quality of the query engine service on the cloud.
