  eBay's experimentation platform runs hundreds of A/B tests on any given day.
The platform integrates with the tracking infrastructure and customer
experience servers, provides the sampling service for experiments, and has the
responsibility to monitor the progress of each A/B test. There are many
challenges especially when it is required to ensure experiment quality at the
large scale. We discuss two automated test quality monitoring processes and
methodologies, namely randomization validation using population stability index
(PSI) and sample ratio mismatch (a.k.a. sample delta) detection using
sequential analysis. The automated processes assist the experimentation
platform to run high quality and trustworthy tests not only effectively on a
large scale, but also efficiently by minimizing false positive monitoring
alarms to experimenters.
