  Convex estimators such as the Lasso, the matrix Lasso and the group Lasso
have been studied extensively in the last two decades, demonstrating great
success in both theory and practice. Two quantities are introduced, the noise
barrier and the large scale bias, that provides insights on the performance of
these convex regularized estimators. It is now well understood that the Lasso
achieves fast prediction rates, provided that the correlations of the design
satisfy some Restricted Eigenvalue or Compatibility condition, and provided
that the tuning parameter is large enough. Using the two quantities introduced
in the paper, we show that the compatibility condition on the design matrix is
actually unavoidable to achieve fast prediction rates with the Lasso. The Lasso
must incur a loss due to the correlations of the design matrix, measured in
terms of the compatibility constant. This results holds for any design matrix,
any active subset of covariates, and any tuning parameter. It is now well known
that the Lasso enjoys a dimension reduction property: the prediction error is
of order $\lambda\sqrt k$ where $k$ is the sparsity; even if the ambient
dimension $p$ is much larger than $k$. Such results require that the tuning
parameters is greater than some universal threshold. We characterize sharp
phase transitions for the tuning parameter of the Lasso around a critical
threshold dependent on $k$. If $\lambda$ is equal or larger than this critical
threshold, the Lasso is minimax over $k$-sparse target vectors. If $\lambda$ is
equal or smaller than critical threshold, the Lasso incurs a loss of order
$\sigma\sqrt k$ --which corresponds to a model of size $k$-- even if the target
vector has fewer than $k$ nonzero coefficients. Remarkably, the lower bounds
obtained in the paper also apply to random, data-driven tuning parameters. The
results extend to convex penalties beyond the Lasso.
