  In this paper, we consider the uncertainty quantification problem for
regression models. Specifically, we consider an individual calibration
objective for characterizing the quantiles of the prediction model. While such
an objective is well-motivated from downstream tasks such as newsvendor cost,
the existing methods have been largely heuristic and lack of statistical
guarantee in terms of individual calibration. We show via simple examples that
the existing methods focusing on population-level calibration guarantees such
as average calibration or sharpness can lead to harmful and unexpected results.
We propose simple nonparametric calibration methods that are agnostic of the
underlying prediction model and enjoy both computational efficiency and
statistical consistency. Our approach enables a better understanding of the
possibility of individual calibration, and we establish matching upper and
lower bounds for the calibration error of our proposed methods. Technically,
our analysis combines the nonparametric analysis with a covering number
argument for parametric analysis, which advances the existing theoretical
analyses in the literature of nonparametric density estimation and quantile
bandit problems. Importantly, the nonparametric perspective sheds new
theoretical insights into regression calibration in terms of the curse of
dimensionality and reconciles the existing results on the impossibility of
individual calibration. To our knowledge, we make the first effort to reach
both individual calibration and finite-sample guarantee with minimal
assumptions in terms of conformal prediction. Numerical experiments show the
advantage of such a simple approach under various metrics, and also under
covariates shift. We hope our work provides a simple benchmark and a starting
point of theoretical ground for future research on regression calibration.
