  A common dilemma encountered by many upon implementing an optimization method
or experiment, whether it be a reinforcement learning algorithm, or A/B
testing, is deciding on what metric to optimize for. Very often short-term
metrics, which are easier to measure are chosen over long term metrics which
have undesirable time considerations and often a more complex calculation. In
this paper, we argue the importance of choosing a metrics that focuses on long
term effects. With this comes the necessity in the ability to measure
significant differences between groups relatively early. We present here an
efficient methodology for early detection of lifetime differences between
groups based on bootstrap hypothesis testing of the lifetime forecast of the
response. We present an application of this method in the domain of online
advertising and we argue that approach not only allows one to focus on the
ultimate metric of importance but also provides a means of accelerating the
testing period.
