Obtaining the magnetic properties of non-resolved structures in the solar photosphere is always challenging and problems arise because the inversion is carried out through the numerical minimization of a merit function that depends on the proposed model. We investigate the reliability of inversions in which the stray-light contamination is obtained from the same observations as a local average. In this case, we show that it is fundamental to include the covariance between the observed Stokes profiles and the stray-light contamination. The ensuing modified merit function of the inversion process penalizes large stray-light contaminations simply because of the presence of positive correlations between the observables and the stray light, fundamentally produced by spatially variable systematics. We caution that if the wrong merit function is used, artificially large stray-light contaminations might be inferred. Since this effect disappears if the stray-light contamination is obtained as an average over the full field of view, we recommend taking into account stray-light contamination using a global approach.