The mass and structural assembly of galaxies is a matter of intense debate. Current theoretical models predict the existence of a linear relationship between galaxy size (Re) and the host dark matter halo virial radius (Rh). By making use of semi-empirical models compared to the size distributions of central galaxies from the Sloan Digital Sky Survey, we provide robust constraints on the normalization and scatter of the Re-Rh relation. We explore the parameter space of models in which the Re-Rh relation is mediated by either the spin parameter or the concentration of the host halo, or a simple constant the nature of which is in principle unknown. We find that the data require extremely tight relations for both early-type and late-type galaxies (ETGs, LTGs), especially for more massive galaxies. These constraints challenge models based solely on angular momentum conservation, which predict significantly wider distributions of galaxy sizes and no trend with stellar mass, if taken at face value. We discuss physically motivated alterations to the original models that bring the predictions into better agreement with the data. We argue that the measured tight size distributions of SDSS disc galaxies can be reproduced by semi-empirical models in which the Re-Rh connection is mediated by the stellar specific angular momenta jstar. We find that current cosmological models of galaxy formation broadly agree with our constraints for LTGs, and justify the strong link between Re and jstar that we propose, however the tightness of the Re-Rh relation found in such ab initio theoretical models for ETGs is in tension with our semi-empirical findings.