The interpretation of local helioseismic measurements of sunspots has long been a challenge, since waves propagating through sunspots are potentially affected by both mode conversion and changes in the thermal structure of the spots. We carry out numerical simulations of wave propagation through a variety of models which alternately isolate either the thermal or magnetic structure of the sunspot or include both of these. We find that helioseismic holography measurements made from the resulting simulated wavefields show qualitative agreement with observations of real sunspots. Using insight from ray theory, we find that travel-time shifts in the thermal (non-magnetic) sunspot model are primarily produced by changes in the wave path due to the Wilson depression rather than variations in the wave speed. This shows that inversions for the subsurface structure of sunspots must account for local changes in the density. In some ranges of horizontal phase speed and frequency there is agreement (within the noise level of the measurements) between the travel times measured in the full magnetic sunspot model and the thermal model. If this conclusion proves to be robust for a wide range of models, it suggests a path towards inversions for sunspot structure. This research has been funded by the Spanish MINECO through grant AYA2014-55078-P, by the NASA Heliophysics Division through NNX14AD42G and NNH12CF23C, and the NSF Solar Terrestrial program through AGS-1127327.