An observational determination of the three-dimensional magnetic and thermal structure of a sunspot is presented. It has been obtained through the application of the SIR inversion technique (Stokes Inversion based on Response functions) on a low-noise, full Stokes profile two-dimensional map of the sunspot as observed with the Advanced Stokes Polarimeter. As a result of the inversion, maps of the magnetic field strength, B, zenith angle, γ, azimuth, χ, and temperature, T, over 25 layers at given optical depths (i.e., an optical tomography) are obtained, of which those between logτ5=0 and logτ5=-2.8 are considered to provide accurate information on the physical parameters. All over the penumbra γ increases with depth, while B is larger at the bottom layers of the inner penumbra (as in the umbra) but larger at the top layers of the outer penumbra (as in the canopy). The corrugation of the penumbral magnetic field already observed by other authors has been confirmed by our different inversion technique. Such a corrugation is especially evident in the zenith angle maps of the intermediate layers, featuring the presence of the so-called spines that we further characterize: spines are warmer and have a less inclined magnetic field than the spaces between them and tend to have a smaller gradient of γ with optical depth over the entire penumbra, but with a field strength which is locally stronger in the middle penumbra and locally weaker in the outer penumbra and beyond in the canopy. In the lower layers of these external parts of the sunspot, most of the field lines are seen to return to the solar surface, a result that is closely connected with the Evershed effect (e.g., Westendorp et al., the third paper in this series). The Stokes V net area asymmetry map as well as the average B, γ, and T radial distributions (and that of the line-of-sight velocities; see the third paper in this series) show a border between an inner and an outer penumbra with different three-dimensional structure. We suggest that it is in this middle zone where most of a new family of penumbral flux tubes (some of them with Evershed flow) emerge interlaced (both horizontally and vertically) among themselves and with the ``background'' magnetic field of the penumbra. The interlacing along the line of sight is witnessed by the indication of many points in the outer penumbra showing rapid transitions with height between two structures, one with very weak and inclined magnetic field at the bottom of the photosphere and the other with a stronger and less inclined magnetic field. Over the whole penumbra, and at all optical layers, a constant but weak deviation from radiality of some 5° is detected for the azimuth of the vector magnetic field, which may be in agreement with former detections but which is not significantly higher than the size of the errors for this parameter.