  Explicit finite-sample statistical guarantees on model performance are an
important ingredient in responsible machine learning. Previous work has focused
mainly on bounding either the expected loss of a predictor or the probability
that an individual prediction will incur a loss value in a specified range.
However, for many high-stakes applications, it is crucial to understand and
control the dispersion of a loss distribution, or the extent to which different
members of a population experience unequal effects of algorithmic decisions. We
initiate the study of distribution-free control of statistical dispersion
measures with societal implications and propose a simple yet flexible framework
that allows us to handle a much richer class of statistical functionals beyond
previous work. Our methods are verified through experiments in toxic comment
detection, medical imaging, and film recommendation.
