Following our recent report of a strong Li lambda 6708 resonance line in the secondary of the black hole candidate V404 Cyg, we present here its detection in the quiescent spectrum of the neutron star binary Cen X-4. We obtain, via a spectral synthesis Local Thermodynamic Equilibrium (LTE) analysis, a surface lithium abundance in the K-type secondary of log N(Li) = 3.3 +/- 0.4. Taking into account non-LTE effects reduces the abundance by 0.2 dex. Using the same analysis, lithium abundances are also derived for the secondaries of the black hole binaries A0620-00, where a lithium detection has very recently been reported, and V404 Cyg. High lithium abundances appear to be a common feature of late-type secondaries in neutron star and black hole binaries, a result totally unexpected mainly because of post-main-sequence dilution and the mass transfer history of these stars. Convective mixing may cause additional depletion if lithium is taken to layers where it is burnt. In order to counteract the depletion it seems necessary to invoke a process of lithium production. Several mechanisms are plausible: for instance, lithium synthesis in a supernova explosion of the compact primary's progenitor, or an on-going process such as alpha-alpha reactions during the repeated strong outbursts that characterize transient X-ray binaries. The spallative mechanism can be tested by future high-resolution gamma-ray and optical spectroscopic observations. Implications on the Li enrichment of the Galactic interstellar medium are briefly discussed.