  Fr\'echet means, conceptually appealing, generalize the Euclidean expectation
to general metric spaces. We explore how well Fr\'echet means can be estimated
from independent and identically distributed samples and uncover a fundamental
limitation: In the vicinity of a probability distribution $P$ with nonunique
means, independent of sample size, it is not possible to uniformly estimate
Fr\'echet means below a precision determined by the diameter of the set of
Fr\'echet means of $P$. Implications were previously identified for empirical
plug-in estimators as part of the phenomenon \emph{finite sample smeariness}.
Our findings thus confirm inevitable statistical challenges in the estimation
of Fr\'echet means on metric spaces for which there exist distributions with
nonunique means. Illustrating the relevance of our lower bound, examples of
extrinsic, intrinsic, Procrustes, diffusion and Wasserstein means showcase
either deteriorating constants or slow convergence rates of empirical Fr\'echet
means for samples near the regime of nonunique means.
