Context. The stability of mass transfer through Roche-lobe overflow plays a key role in shaping the outcome of binary interactions. However, the criterion for mass transfer stability remains one of the main open questions in the theory of binary evolution. Aims. Our aim was to close this knowledge gap by developing a mass transfer stability criterion that accounts for mass and angular momentum loss, and implemented it in the population synthesis code SEVN. We assessed its impact relative to the standard formalism used in SEVN, using blue straggler stars and binary compact objects as illustrative cases. Methods. We derived an expression for the response of the Roche-lobe radius to mass loss in the general case where the mass and angular momentum of the system are not conserved. On the basis of this formulation, we constructed a new mass transfer stability criterion that modifies the standard approach only through the Roche-lobe response term. We performed population synthesis simulations with the SEVN code to explore the effects of the new stability criterion. Results. The new criterion allows stable mass transfer in binaries with higher donor-to-accretor mass ratios, leading to an overall increase in the predicted number of blue stragglers and promoting their formation in wider orbits. This contributes to reconciling the differences between theory and observations. For binary compact objects, the impact of the new stability prescription varies across system types, with the strongest effects occurring in binaries containing at least one neutron star. In particular, for low mass transfer efficiency, the new prescription enhances the contribution of channels involving stable mass transfer and leads to a larger number of systems, including gravitational wave progenitors. Conclusions. The inclusion of a new, simple, yet more consistent prescription for mass transfer stability has proven that refining this criterion can significantly improve our understanding of the formation channels of specific stellar populations.