  We give a general result concerning the rates of convergence of penalized
empirical risk minimizers (PERM) in the regression model. Then, we consider the
problem of agnostic learning of the regression, and give in this context an
oracle inequality and a lower bound for PERM over a finite class. These results
hold for a general multivariate random design, the only assumption being the
compactness of the support of its law (allowing discrete distributions for
instance). Then, using these results, we construct adaptive estimators. We
consider as examples adaptive estimation over anisotropic Besov spaces or
reproductive kernel Hilbert spaces. Finally, we provide an empirical evidence
that aggregation leads to more stable estimators than more standard
cross-validation or generalized cross-validation methods for the selection of
the smoothing parameter, when the number of observation is small.
