  We study theoretical properties of a broad class of regularized algorithms
with vector-valued output. These spectral algorithms include kernel ridge
regression, kernel principal component regression, various implementations of
gradient descent and many more. Our contributions are twofold. First, we
rigorously confirm the so-called saturation effect for ridge regression with
vector-valued output by deriving a novel lower bound on learning rates; this
bound is shown to be suboptimal when the smoothness of the regression function
exceeds a certain level. Second, we present the upper bound for the finite
sample risk general vector-valued spectral algorithms, applicable to both
well-specified and misspecified scenarios (where the true regression function
lies outside of the hypothesis space) which is minimax optimal in various
regimes. All of our results explicitly allow the case of infinite-dimensional
output variables, proving consistency of recent practical applications.
