We report optical and infrared observations of the X-ray source CXOGBS J174623.5-310550. This Galactic object was identified as a potential quiescent low-mass X-ray binary accreting from an M-type donor on the basis of optical spectroscopy and the broad H α emission line. The analysis of X-shooter spectroscopy covering three consecutive nights supports an M2/3-type spectral classification. Neither radial velocity variations nor rotational broadening is detected in the photospheric lines. No periodic variability is found in I- and r'-band light curves. We derive r' = 20.8, I = 19.2, and Ks ≈ 16.6 for the optical and infrared counterparts with the M-type star contributing {≈ }90{{ per cent}} to the I-band light. We estimate its distance to be 1.3-1.8 kpc. The lack of radial velocity variations implies that the M-type star is not the donor star in the X-ray binary. This could be an interloper or the outer body in a hierarchical triple. We constrain the accreting binary to be a ≲2.2 h orbital period eclipsing cataclysmic variable or a low-mass X-ray binary lying in the foreground of the Galactic bulge.