Context. Radial velocity (RV) measurements from near-infrared spectra have become a potentially powerful tool to search for planets around cool stars and sub-stellar objects. As part of a large survey to characterize M-dwarfs using NIRSPEC at Keck II, we obtained spectra of eight late M-dwarfs (spectral types M5.0-M8.0) during two or more observing epochs per target. These spectra were taken with intermediate spectral resolving powers (R ~ 20 000) in the J-band. Aims: We search for relative RV variability in these late M-dwarfs and test the NIRSPEC capability of detecting short-period brown dwarf and massive planetary companions around low-mass stars in the J-band (≈1.25 μm). Additionally, we reanalyzed the data of the M8-type star vB10 (one of our targets), which had been presented in another article. Methods: To achieve a precise RV measurement stability, the NIRSPEC spectra were self-calibrated by making use of the telluric absorption lines, which are present in the observed spectra and were used as a long-term stable reference. In the modeling process a multi-parameter χ2-optimization was employed to generate an accurate description of the observation. The telluric lines allowed us to model the instrumental profile of the spectrograph and the determination of the Doppler shift of the stellar absorption lines. Results: For the entire M-dwarf sample, we found no evidence of relative RV variations induced by a short-period brown dwarf or massive planetary companion. The typical RV precision of the measurements is between 180 and 300 m s-1, which is sufficient to detect hot Neptunes around M-dwarfs. Moreover, we found that the spurious RV shift detected previously of the star VB10 was caused by asymmetries in the instrumental profile between different observing epochs, which were not taken into account in the previous analysis.