  We study the excess minimum risk in statistical inference, defined as the
difference between the minimum expected loss in estimating a random variable
from an observed feature vector and the minimum expected loss in estimating the
same random variable from a transformation (statistic) of the feature vector.
After characterizing lossless transformations, i.e., transformations for which
the excess risk is zero for all loss functions, we construct a partitioning
test statistic for the hypothesis that a given transformation is lossless and
show that for i.i.d. data the test is strongly consistent. More generally, we
develop information-theoretic upper bounds on the excess risk that uniformly
hold over fairly general classes of loss functions. Based on these bounds, we
introduce the notion of a delta-lossless transformation and give sufficient
conditions for a given transformation to be universally delta-lossless.
Applications to classification, nonparametric regression, portfolio strategies,
information bottleneck, and deep learning, are also surveyed.
