  With the growing needs of online A/B testing to support the innovation in
industry, the opportunity cost of running an experiment becomes non-negligible.
Therefore, there is an increasing demand for an efficient continuous monitoring
service that allows early stopping when appropriate. Classic statistical
methods focus on hypothesis testing and are mostly developed for traditional
high-stake problems such as clinical trials, while experiments at online
service companies typically have very different features and focuses. Motivated
by the real needs, in this paper, we introduce a novel framework that we
developed in Amazon to maximize customer experience and control opportunity
cost. We formulate the problem as a Bayesian optimal sequential decision making
problem that has a unified utility function. We discuss extensively practical
design choices and considerations. We further introduce how to solve the
optimal decision rule via Reinforcement Learning and scale the solution. We
show the effectiveness of this novel approach compared with existing methods
via a large-scale meta-analysis on experiments in Amazon.
