  The practice of continuous deployment has enabled companies to reduce
time-to-market by increasing the rate at which software can be deployed.
However, deploying more frequently bears the risk that occasionally defective
changes are released. For Internet companies, this has the potential to degrade
the user experience and increase user abandonment. Therefore, quality control
gates are an important component of the software delivery process. These are
used to build confidence in the reliability of a release or change. Towards
this end, a common approach is to perform a canary test to evaluate new
software under production workloads. Detecting defects as early as possible is
necessary to reduce exposure and to provide immediate feedback to the
developer. We present a statistical framework for rapidly detecting regressions
in software deployments. Our approach is based on sequential tests of
stochastic order and of equality in distribution. This enables canary tests to
be continuously monitored, permitting regressions to be rapidly detected while
strictly controlling the false detection probability throughout. The utility of
this approach is demonstrated based on two case studies at Netflix.
