Using data obtained with the ESO-Very Large Telescope, we have performed a deep optical and NIR photometric study of the fields of three Accreting Millisecond X-ray Pulsars (XTE J1814-338, XTE J0929-314 and XTE J1807-294) during quiescence in order to look for the presence of a variable counterpart. If suitable candidates are found, we also carried out optical spectroscopy. We present here the first multi-band (VR) detection of the optical counterpart of XTE J1814-338 in quiescence together with its optical spectrum. The optical light curve shows variability in both bands consistent with a sinusoidal modulation at the known 4.3 hr orbital period. The marginal detection of the very faint counterpart of XTE J0929-314 and deep upper limits for the optical/NIR counterpart of XTE J1807-294 are also reported. We also briefly discuss the results reported in the literature for the optical/NIR counterpart of XTE J1751-305. Our findings are consistent with AMXPs being systems containing an old, weakly magnetized neutron star, reactivated as a millisecond radio pulsar during quiescence which irradiates the low-mass companion star. The absence of type I X-ray bursts and of hydrogen and helium lines in outburst spectra of ultra-compact (Porb<1 hr) AMXPs suggests that the companion stars are likely evolved dwarf stars.