The effective or half-light radius has become a very popular choice for characterising galaxy size, but it measures the concentration of light within galaxies and thus does not capture our intuitive definition of size which is related to the edge or boundary of objects. We define a new physically motivated measurement of size for galaxies, R1, based on the expected location of the gas density threshold for star formation, for which we use the stellar mass density contour at 1 Msun pc-2 as a proxy. With this new size measure, the intrinsic scatter of the global stellar mass (Mstar)-size relation (explored over five orders of magnitude in stellar mass) decreases to ∼0.06 dex: 2.5 times smaller than the scatter measured using the effective radius (∼0.15 dex). Galaxies with Mstar > 1011 Msun show a different slope with stellar mass than less massive galaxies, which is suggestive of a larger gas density threshold for star formation at the epoch when their star formation peaks. We go on to investigate the nature of faint ultra-diffuse galaxies (UDGs) when compared to dwarfs and Milky Way-like galaxies. Using our new R1 measure instead of the effective radius and considering the sizes and stellar mass density profiles of UDGs and regular dwarfs, we find that the UDGs have sizes that are within the size range of dwarfs and, on average, UDGs are ten times smaller than Milky Way-like galaxies. These results show that the use of size estimators sensitive to the concentration of light can lead to misleading results.