  We consider the problem of estimating a low rank covariance function $K(t,u)$
of a Gaussian process $S(t), t\in [0,1]$ based on $n$ i.i.d. copies of $S$
observed in a white noise. We suggest a new estimation procedure adapting
simultaneously to the low rank structure and the smoothness of the covariance
function. The new procedure is based on nuclear norm penalization and exhibits
superior performances as compared to the sample covariance function by a
polynomial factor in the sample size $n$. Other results include a minimax lower
bound for estimation of low-rank covariance functions showing that our
procedure is optimal as well as a scheme to estimate the unknown noise variance
of the Gaussian process.
