  It is a common phenomenon that for high-dimensional and nonparametric
statistical models, rate-optimal estimators balance squared bias and variance.
Although this balancing is widely observed, little is known whether methods
exist that could avoid the trade-off between bias and variance. We propose a
general strategy to obtain lower bounds on the variance of any estimator with
bias smaller than a prespecified bound. This shows to which extent the
bias-variance trade-off is unavoidable and allows to quantify the loss of
performance for methods that do not obey it. The approach is based on a number
of abstract lower bounds for the variance involving the change of expectation
with respect to different probability measures as well as information measures
such as the Kullback-Leibler or $\chi^2$-divergence. In a second part of the
article, the abstract lower bounds are applied to several statistical models
including the Gaussian white noise model, a boundary estimation problem, the
Gaussian sequence model and the high-dimensional linear regression model. For
these specific statistical applications, different types of bias-variance
trade-offs occur that vary considerably in their strength. For the trade-off
between integrated squared bias and integrated variance in the Gaussian white
noise model, we propose to combine the general strategy for lower bounds with a
reduction technique. This allows us to reduce the original problem to a lower
bound on the bias-variance trade-off for estimators with additional symmetry
properties in a simpler statistical model. In the Gaussian sequence model,
different phase transitions of the bias-variance trade-off occur. Although
there is a non-trivial interplay between bias and variance, the rate of the
squared bias and the variance do not have to be balanced in order to achieve
the minimax estimation rate.
