  The regularization path of the Lasso can be shown to be piecewise linear,
making it possible to "follow" and explicitly compute the entire path. We
analyze in this paper this popular strategy, and prove that its worst case
complexity is exponential in the number of variables. We then oppose this
pessimistic result to an (optimistic) approximate analysis: We show that an
approximate path with at most O(1/sqrt(epsilon)) linear segments can always be
obtained, where every point on the path is guaranteed to be optimal up to a
relative epsilon-duality gap. We complete our theoretical analysis with a
practical algorithm to compute these approximate paths.
