Aims: The aim of the project is to define metallicity/gravity/temperature scales for different spectral types of metal-poor M dwarfs. Methods: We obtained intermediate-resolution ultraviolet (R ˜ 3300), optical (R ˜ 5400), and near-infrared (R ˜ 3900) spectra of 43 M subdwarfs (sdM), extreme subdwarfs (esdM), and ultra-subdwarfs (usdM) with the X-shooter spectrograph on the European Southern Observatory Very Large Telescope. We compared our atlas of spectra to the latest BT-Settl synthetic spectral energy distribution over a wide range of metallicities, gravities, and effective temperatures to infer the physical properties for the whole M dwarf sequence (M0-M9.5) at sub-solar metallicities and constrain the latest atmospheric models. Results: The BT-Settl models accurately reproduce the observed spectra across the 450-2500 nm wavelength range except for a few regions. We find that the best fits are obtained for gravities of log (g) = 5.0-5.5 for the three metal classes. We infer metallicities of [Fe/H] = -0.5, -1.5, and -2.0 ± 0.5 dex and effective temperatures of 3700-2600 K, 3800-2900 K, and 3700-2900 K for subdwarfs, extreme subdwarfs, and ultra-subdwarfs, respectively. Metal-poor M dwarfs tend to be warmer by about 200 ± 100 K and exhibit higher gravity than their solar-metallicity counterparts. We derive abundances of several elements (Fe, Na, K, Ca, Ti) for our sample but cannot describe their atmospheres with a single metallicity parameter. Our metallicity scale expands the current scales available for mildly metal-poor planet-host low-mass stars. Our compendium of moderate-resolution spectra covering the 0.45-2.5 micron range represents an important resource for large-scale surveys and space missions to come. All observed spectra are available at the CDS via anonymous ftp to http://cdsarc.u-strasbg.fr (ftp://130.79.128.5) or via http://cdsarc.u-strasbg.fr/viz-bin/qcat?J/A+A/628/A61Based on observations collected at the European Southern Observatory, Chile, under programmes 089.C-0140(A), 091.C-0264(A), 092.D-0600(A), and 093.C-0610(A).