  A/B tests are typically analyzed via frequentist p-values and confidence
intervals; but these inferences are wholly unreliable if users endogenously
choose samples sizes by *continuously monitoring* their tests. We define
*always valid* p-values and confidence intervals that let users try to take
advantage of data as fast as it becomes available, providing valid statistical
inference whenever they make their decision. Always valid inference can be
interpreted as a natural interface for a sequential hypothesis test, which
empowers users to implement a modified test tailored to them. In particular, we
show in an appropriate sense that the measures we develop tradeoff sample size
and power efficiently, despite a lack of prior knowledge of the user's relative
preference between these two goals. We also use always valid p-values to obtain
multiple hypothesis testing control in the sequential context. Our methodology
has been implemented in a large scale commercial A/B testing platform to
analyze hundreds of thousands of experiments to date.
