  Tech companies (e.g., Google or Facebook) often use randomized online
experiments and/or A/B testing primarily based on the average treatment effects
to compare their new product with an old one. However, it is also critically
important to detect qualitative treatment effects such that the new one may
significantly outperform the existing one only under some specific
circumstances. The aim of this paper is to develop a powerful testing procedure
to efficiently detect such qualitative treatment effects. We propose a scalable
online updating algorithm to implement our test procedure. It has three
novelties including adaptive randomization, sequential monitoring, and online
updating with guaranteed type-I error control. We also thoroughly examine the
theoretical properties of our testing procedure including the limiting
distribution of test statistics and the justification of an efficient bootstrap
method. Extensive empirical studies are conducted to examine the finite sample
performance of our test procedure.
