  If you are predicting the label $y$ of a new object with $\hat y$, how
confident are you that $y = \hat y$? Conformal prediction methods provide an
elegant framework for answering such question by building a $100 (1 -
\alpha)\%$ confidence region without assumptions on the distribution of the
data. It is based on a refitting procedure that parses all the possibilities
for $y$ to select the most likely ones. Although providing strong coverage
guarantees, conformal set is impractical to compute exactly for many regression
problems. We propose efficient algorithms to compute conformal prediction set
using approximated solution of (convex) regularized empirical risk
minimization. Our approaches rely on a new homotopy continuation technique for
tracking the solution path with respect to sequential changes of the
observations. We also provide a detailed analysis quantifying its complexity.
