  Common statistical measures of uncertainty such as $p$-values and confidence
intervals quantify the uncertainty due to sampling, that is, the uncertainty
due to not observing the full population. However, sampling is not the only
source of uncertainty. In practice, distributions change between locations and
across time. This makes it difficult to gather knowledge that transfers across
data sets. We propose a measure of instability that quantifies the
distributional instability of a statistical parameter with respect to
Kullback-Leibler divergence, that is, the sensitivity of the parameter under
general distributional perturbations within a Kullback-Leibler divergence ball.
In addition, we quantify the instability of parameters with respect to
directional or variable-specific shifts. Measuring instability with respect to
directional shifts can be used to detect the type of shifts a parameter is
sensitive to. We discuss how such knowledge can inform data collection for
improved estimation of statistical parameters under shifted distributions. We
evaluate the performance of the proposed measure on real data and show that it
can elucidate the distributional instability of a parameter with respect to
certain shifts and can be used to improve estimation accuracy under shifted
distributions.
