  Inference for prediction errors is critical in time series forecasting
pipelines. However, providing statistically meaningful uncertainty intervals
for prediction errors remains relatively under-explored. Practitioners often
resort to forward cross-validation (FCV) for obtaining point estimators and
constructing confidence intervals based on the Central Limit Theorem (CLT). The
naive version assumes independence, a condition that is usually invalid due to
time correlation. These approaches lack statistical interpretations and
theoretical justifications even under stationarity.
  This paper systematically investigates uncertainty intervals for prediction
errors in time series forecasting. We first distinguish two key inferential
targets: the stochastic test error over near future data points, and the
expected test error as the expectation of the former. The stochastic test error
is often more relevant in applications needing to quantify uncertainty over
individual time series instances. To construct prediction intervals for the
stochastic test error, we propose the quantile-based forward cross-validation
(QFCV) method. Under an ergodicity assumption, QFCV intervals have
asymptotically valid coverage and are shorter than marginal empirical
quantiles. In addition, we also illustrate why naive CLT-based FCV intervals
fail to provide valid uncertainty intervals, even with certain corrections. For
non-stationary time series, we further provide rolling intervals by combining
QFCV with adaptive conformal prediction to give time-average coverage
guarantees. Overall, we advocate the use of QFCV procedures and demonstrate
their coverage and efficiency through simulations and real data examples.
