  Uncertainty quantification is crucial for assessing the predictive ability of
AI algorithms. Much research has been devoted to describing the predictive
distribution (PD) $F(y|\mathbf{x})$ of a target variable $y \in \mathbb{R}$
given complex input features $\mathbf{x} \in \mathcal{X}$. However,
off-the-shelf PDs (from, e.g., normalizing flows and Bayesian neural networks)
often lack conditional calibration with the probability of occurrence of an
event given input $\mathbf{x}$ being significantly different from the predicted
probability. Current calibration methods do not fully assess and enforce
conditionally calibrated PDs. Here we propose \texttt{Cal-PIT}, a method that
addresses both PD diagnostics and recalibration by learning a single
probability-probability map from calibration data. The key idea is to regress
probability integral transform scores against $\mathbf{x}$. The estimated
regression provides interpretable diagnostics of conditional coverage across
the feature space. The same regression function morphs the misspecified PD to a
re-calibrated PD for all $\mathbf{x}$. We benchmark our corrected prediction
bands (a by-product of corrected PDs) against oracle bands and state-of-the-art
predictive inference algorithms for synthetic data. We also provide results for
two applications: (i) probabilistic nowcasting given sequences of satellite
images, and (ii) conditional density estimation of galaxy distances given
imaging data (so-called photometric redshift estimation). Our code is available
as a Python package https://github.com/lee-group-cmu/Cal-PIT .
