  Firms implementing digital advertising campaigns face a complex problem in
determining the right match between their advertising creatives and target
audiences. Typical solutions to the problem have leveraged non-experimental
methods, or used "split-testing" strategies that have not explicitly addressed
the complexities induced by targeted audiences that can potentially overlap
with one another. This paper presents an adaptive algorithm that addresses the
problem via online experimentation. The algorithm is set up as a contextual
bandit and addresses the overlap issue by partitioning the target audiences
into disjoint, non-overlapping sub-populations. It learns an optimal creative
display policy in the disjoint space, while assessing in parallel which
creative has the best match in the space of possibly overlapping target
audiences. Experiments show that the proposed method is more efficient compared
to naive "split-testing" or non-adaptive "A/B/n" testing based methods. We also
describe a testing product we built that uses the algorithm. The product is
currently deployed on the advertising platform of JD.com, an eCommerce company
and a publisher of digital ads in China.
