  Online A/B testing plays a critical role in the high-tech industry to guide
product development and accelerate innovation. It performs a null hypothesis
statistical test to determine which variant is better. However, a typical A/B
test presents two problems: (i) a fixed-horizon framework inflates the
false-positive errors under continuous monitoring; (ii) the homogeneous effects
assumption fails to identify a subgroup with a beneficial treatment effect. In
this paper, we propose a sequential test for subgroup treatment effects based
on value difference, named SUBTLE, to address these two problems
simultaneously. The SUBTLE allows the experimenters to "peek" at the results
during the experiment without harming the statistical guarantees. It assumes
heterogeneous treatment effects and aims to test if some subgroup of the
population will benefit from the investigative treatment. If the testing result
indicates the existence of such a subgroup, a subgroup will be identified using
a readily available estimated optimal treatment rule. We examine the empirical
performance of our proposed test on both simulations and a real dataset. The
results show that the SUBTLE has high detection power with controlled type I
error at any time, is more robust to noise covariates, and can achieve early
stopping compared with the corresponding fixed-horizon test.
