  Conducting experiments with objectives that take significant delays to
materialize (e.g. conversions, add-to-cart events, etc.) is challenging.
Although the classical "split sample testing" is still valid for the delayed
feedback, the experiment will take longer to complete, which also means
spending more resources on worse-performing strategies due to their fixed
allocation schedules. Alternatively, adaptive approaches such as "multi-armed
bandits" are able to effectively reduce the cost of experimentation. But these
methods generally cannot handle delayed objectives directly out of the box.
This paper presents an adaptive experimentation solution tailored for delayed
binary feedback objectives by estimating the real underlying objectives before
they materialize and dynamically allocating variants based on the estimates.
Experiments show that the proposed method is more efficient for delayed
feedback compared to various other approaches and is robust in different
settings. In addition, we describe an experimentation product powered by this
algorithm. This product is currently deployed in the online experimentation
platform of JD.com, a large e-commerce company and a publisher of digital ads.
