  The mass and structural assembly of galaxies is a matter of intense debate.
Current theoretical models predict the existence of a linear relationship
between galaxy size ($R_e$) and the host dark matter halo virial radius
($R_h$).\\ By making use of semi-empirical models compared to the size
distributions of central galaxies from the Sloan Digital Sky Survey, we provide
robust constraints on the normalization and scatter of the $R_e-R_h$ relation.
We explore the parameter space of models in which the $R_e-R_h$ relation is
mediated by either the spin parameter or the concentration of the host halo, or
a simple constant the nature of which is in principle unknown. We find that the
data require extremely tight relations for both early-type and late-type
galaxies (ETGs,LTGs), especially for more massive galaxies. These constraints
challenge models based solely on angular momentum conservation, which predict
significantly wider distributions of galaxy sizes and no trend with stellar
mass, if taken at face value. We discuss physically-motivated alterations to
the original models that bring the predictions into better agreement with the
data. We argue that the measured tight size distributions of SDSS disk galaxies
can be reproduced by semi-empirical models in which the $R_e-R_h$ connection is
mediated by the \emph{stellar} specific angular momenta $j_{star}.$ We find
that current cosmological models of galaxy formation broadly agree with our
constraints for LTGs, and justify the strong link between $R_e$ and $j_{star}$
that we propose, however the tightness of the $R_e-R_h$ relation found in such
ab-initio theoretical models for ETGs is in tension with our semi-empirical
findings.
