  A/B testers conducting large-scale tests prioritize lifts and want to be able
to control false rejections of the null. This work develops a
decision-theoretic framework for maximizing profits subject to false discovery
rate (FDR) control. We build an empirical Bayes solution for the problem via
the greedy knapsack approach. We derive an oracle rule based on ranking the
ratio of expected lifts and the cost of wrong rejections using the local false
discovery rate (lfdr) statistic. Our oracle decision rule is valid and optimal
for large-scale tests. Further, we establish asymptotic validity for the
data-driven procedure and demonstrate finite-sample validity in experimental
studies. We also demonstrate the merit of the proposed method over other FDR
control methods. Finally, we discuss an application to actual Optimizely
experiments.
