Context. A significant part of the penumbral magnetic field returns below the surface in the very deep photosphere. For lines in the visible, a large portion of this return field can only be detected indirectly by studying its imprints on strongly asymmetric and three-lobed Stokes V profiles. Infrared lines probe a narrow layer in the very deep photosphere, providing the possibility of directly measuring the orientation of magnetic fields close to the solar surface. Aims: We study the topology of the penumbral magnetic field in the lower photosphere, focusing on regions where it returns below the surface. Methods: We analyzed 71 spectropolarimetric datasets from Hinode and from the GREGOR infrared spectrograph. We inferred the quality and polarimetric accuracy of the infrared data after applying several reduction steps. Techniques of spectral inversion and forward synthesis were used to test the detection algorithm. We compared the morphology and the fractional penumbral area covered by reversed-polarity and three-lobed Stokes V profiles for sunspots at disk center. We determined the amount of reversed-polarity and three-lobed Stokes V profiles in visible and infrared data of sunspots at various heliocentric angles. From the results, we computed center-to-limb variation curves, which were interpreted in the context of existing penumbral models. Results: Observations in visible and near-infrared spectral lines yield a significant difference in the penumbral area covered by magnetic fields of opposite polarity. In the infrared, the number of reversed-polarity Stokes V profiles is smaller by a factor of two than in the visible. For three-lobed Stokes V profiles the numbers differ by up to an order of magnitude.