  We introduce Minimal Achievable Sufficient Statistic (MASS) Learning, a
training method for machine learning models that attempts to produce minimal
sufficient statistics with respect to a class of functions (e.g. deep networks)
being optimized over. In deriving MASS Learning, we also introduce Conserved
Differential Information (CDI), an information-theoretic quantity that - unlike
standard mutual information - can be usefully applied to
deterministically-dependent continuous random variables like the input and
output of a deep network. In a series of experiments, we show that deep
networks trained with MASS Learning achieve competitive performance on
supervised learning and uncertainty quantification benchmarks.
