  Randomized experiments have become the standard method for companies to
evaluate the performance of new products or services. In addition to augmenting
managers' decision-making, experimentation mitigates risk by limiting the
proportion of customers exposed to innovation. Since many experiments are on
customers arriving sequentially, a potential solution is to allow managers to
"peek" at the results when new data becomes available and stop the test if the
results are statistically significant. Unfortunately, peeking invalidates the
statistical guarantees for standard statistical analysis and leads to
uncontrolled type-1 error. Our paper provides valid design-based confidence
sequences, sequences of confidence intervals with uniform type-1 error
guarantees over time for various sequential experiments in an assumption-light
manner. In particular, we focus on finite-sample estimands defined on the study
participants as a direct measure of the incurred risks by companies. Our
proposed confidence sequences are valid for a large class of experiments,
including multi-arm bandits, time series, and panel experiments. We further
provide a variance reduction technique incorporating modeling assumptions and
covariates. Finally, we demonstrate the effectiveness of our proposed approach
through a simulation study and three real-world applications from Netflix. Our
results show that by using our confidence sequence, harmful experiments could
be stopped after only observing a handful of units; for instance, an experiment
that Netflix ran on its sign-up page on 30,000 potential customers would have
been stopped by our method on the first day before 100 observations.
