  Predictive modelling and supervised learning are central to modern data
science. With predictions from an ever-expanding number of supervised black-box
strategies - e.g., kernel methods, random forests, deep learning aka neural
networks - being employed as a basis for decision making processes, it is
crucial to understand the statistical uncertainty associated with these
predictions.
  As a general means to approach the issue, we present an overarching framework
for black-box prediction strategies that not only predict the target but also
their own predictions' uncertainty. Moreover, the framework allows for fair
assessment and comparison of disparate prediction strategies. For this, we
formally consider strategies capable of predicting full distributions from
feature variables, so-called probabilistic supervised learning strategies.
  Our work draws from prior work including Bayesian statistics, information
theory, and modern supervised machine learning, and in a novel synthesis leads
to (a) new theoretical insights such as a probabilistic bias-variance
decomposition and an entropic formulation of prediction, as well as to (b) new
algorithms and meta-algorithms, such as composite prediction strategies,
probabilistic boosting and bagging, and a probabilistic predictive independence
test.
  Our black-box formulation also leads (c) to a new modular interface view on
probabilistic supervised learning and a modelling workflow API design, which we
have implemented in the newly released skpro machine learning toolbox,
extending the familiar modelling interface and meta-modelling functionality of
sklearn. The skpro package provides interfaces for construction, composition,
and tuning of probabilistic supervised learning strategies, together with
orchestration features for validation and comparison of any such strategy - be
it frequentist, Bayesian, or other.
