  We study distribution free, nonparametric prediction bands with a special
focus on their finite sample behavior. First we investigate and develop
different notions of finite sample coverage guarantees. Then we give a new
prediction band estimator by combining the idea of "conformal prediction" (Vovk
et al. 2009) with nonparametric conditional density estimation. The proposed
estimator, called COPS (Conformal Optimized Prediction Set), always has finite
sample guarantee in a stronger sense than the original conformal prediction
estimator. Under regularity conditions the estimator converges to an oracle
band at a minimax optimal rate. A fast approximation algorithm and a data
driven method for selecting the bandwidth are developed. The method is
illustrated first in simulated data. Then, an application shows that the
proposed method gives desirable prediction intervals in an automatic way, as
compared to the classical linear regression modeling.
