We explore whether medium-resolution stellar spectra can be reconstructed from photometric observations, taking advantage of the highly compressible nature of the spectra. We formulate the spectral reconstruction as a least-squares problem with a sparsity constraint. In our test case using data from the Sloan Digital Sky Survey, only three broadband filters are used as input. We demonstrate that reconstruction using three principal components is feasible with these filters, leading to median differences with respect to the original spectrum smaller than 5%. We analyze the effect of uncertainties in the observed magnitudes and find that the available high photometric precision induces very small errors in the reconstruction. This process may facilitate the extraction of purely spectroscopic quantities, such as the overall metallicity, for hundreds of millions of stars for which only photometric information is available, using standard analysis techniques applied to the reconstructed spectra.