Polarized light provides the most reliable source of information at our disposal for diagnosing the physical properties of astrophysical plasmas, including the three-dimensional (3D) structure of the solar atmosphere. Here we formulate and solve the 3D radiative transfer problem of the linear polarization of the solar continuous radiation, which is principally produced by Rayleigh and Thomson scattering. Our approach takes into account not only the anisotropy of the solar continuum radiation but also the symmetry-breaking effects caused by the horizontal atmospheric inhomogeneities produced by the solar surface convection. We show that such symmetry-breaking effects do produce observable signatures in Q/I and U/I, even at the very center of the solar disk where we observe the forward scattering case, but their detection would require obtaining very high resolution linear polarization images of the solar surface. Without spatial and/or temporal resolution U/I ≈ 0 and the only observable quantity is Q/I, whose wavelength variation at a solar disk position close to the limb has been recently determined semi-empirically. Interestingly, our 3D radiative transfer modeling of the polarization of the Sun's continuous spectrum in a well-known 3D hydrodynamical model of the solar photosphere shows remarkable agreement with the semi-empirical determination, significantly better than that obtained via the use of one-dimensional (1D) atmospheric models. Although this result confirms that the above-mentioned 3D model was indeed a suitable choice for our Hanle-effect estimation of the substantial amount of "hidden" magnetic energy that is stored in the quiet solar photosphere, we have found however some small discrepancies whose origin may be due to uncertainties in the semi-empirical data and/or in the thermal and density structure of the 3D model. For this reason, we have paid some attention also to other (more familiar) observables, like the center-limb variation of the continuum intensity, which we have calculated taking into account the scattering contribution to the continuum source function. The overall agreement with the observed center-limb variation turns out to be impressive, but we find a hint that the model's temperature gradients in the continuum-forming layers could be slightly too steep, perhaps because all current simulations of solar surface convection and magnetoconvection compute the radiative flux divergence ignoring the fact that the effective polarizability is not completely negligible, especially in the downward-moving intergranular lane plasma.