  We derive, in the classical framework of Bayesian sensitivity analysis,
optimal lower and upper bounds on posterior values obtained from Bayesian
models that exactly capture an arbitrarily large number of finite-dimensional
marginals of the data-generating distribution and/or that are as close as
desired to the data-generating distribution in the Prokhorov or total variation
metrics; these bounds show that such models may still make the largest possible
prediction error after conditioning on an arbitrarily large number of sample
data measured at finite precision. These results are obtained through the
development of a reduction calculus for optimization problems over measures on
spaces of measures. We use this calculus to investigate the mechanisms that
generate brittleness/robustness and, in particular, we observe that learning
and robustness are antagonistic properties. It is now well understood that the
numerical resolution of PDEs requires the satisfaction of specific stability
conditions. Is there a missing stability condition for using Bayesian inference
in a continuous world under finite information?
