Observations of metal absorption systems in the spectra of distant quasars allow one to constrain a possible variation of the fine-structure constant throughout the history of the Universe. Such a test poses utmost demands on the wavelength accuracy and previous studies were limited by systematics in the spectrograph wavelength calibration. A substantial advance in the field is therefore expected from the new ultra-stable high-resolution spectrograph ESPRESSO, which was recently installed at the VLT. In preparation of the fundamental physics related part of the ESPRESSO GTO program, we present a thorough assessment of the ESPRESSO wavelength accuracy and identify possible systematics at each of the different steps involved in the wavelength calibration process. Most importantly, we compare the default wavelength solution, which is based on the combination of Thorium-Argon arc lamp spectra and a Fabry-Pérot interferometer, to the fully independent calibration obtained from a laser frequency comb. We find wavelength-dependent discrepancies of up to 24 m s-1. This substantially exceeds the photon noise and highlights the presence of different sources of systematics, which we characterize in detail as part of this study. Nevertheless, our study demonstrates the outstanding accuracy of ESPRESSO with respect to previously used spectrographs and we show that constraints of a relative change of the fine-structure constant at the 10-6 level can be obtained with ESPRESSO without being limited by wavelength calibration systematics.