High resolution spectroscopy demands detector operation in an extremely stable mechanical and thermal environment; any tiny perturbation may affect the location of the detected spectral lines, affecting the instrument global performance. The detector stability has been identified currently as one of the limiting factors to improve radial velocity measurements accuracy, at the sub m/s level, and an activity focused in the performance understanding of the detector system and how to improve those has been developed. HARPS detector system has been adopted as baseline and the results obtained in a test campaign are described; next activities in the development to improve detector system performance are highlighted, concluding with the information gathered for setting up radial velocity error budgets for the next generation of ESO high stability and high resolution spectrographs.