  The performance of decision policies and prediction models often deteriorates
when applied to environments different from the ones seen during training. To
ensure reliable operation, we analyze the stability of a system under
distribution shift, which is defined as the smallest change in the underlying
environment that causes the system's performance to deteriorate beyond a
permissible threshold. In contrast to standard tail risk measures and
distributionally robust losses that require the specification of a plausible
magnitude of distribution shift, the stability measure is defined in terms of a
more intuitive quantity: the level of acceptable performance degradation. We
develop a minimax optimal estimator of stability and analyze its convergence
rate, which exhibits a fundamental phase shift behavior. Our characterization
of the minimax convergence rate shows that evaluating stability against large
performance degradation incurs a statistical cost. Empirically, we demonstrate
the practical utility of our stability framework by using it to compare system
designs on problems where robustness to distribution shift is critical.
