Context. Radial velocity (RV) jitter represents an intrinsic limitation on the precision of Doppler searches for exoplanets that can originate from both instrumental and astrophysical sources. Aims. We aim to determine the RV jitter floor in M dwarfs and investigate the stellar properties that lead to RV jitter induced by stellar activity. Methods. We determined the RV jitter in 239 M dwarfs from the CARMENES survey that are predominantly of mid to late spectral type and solar metallicity. We also investigated the correlation between stellar rotation and magnetic fields with RV jitter. Results. The median jitter in the CARMENES sample is 3.1 m s‑1, and it is 2.3 m s‑1 for stars with an upper limit of 2 km s‑1 on their projected rotation velocities. We provide a relation between the stellar equatorial rotation velocity and RV jitter in M dwarfs based on a subsample of 129 well-characterized CARMENES stars. RV jitter induced by stellar rotation dominates for stars with equatorial rotation velocities greater than 1 km s‑1. A jitter floor of 2 m s‑1 dominates in stars with equatorial rotation velocities below 1 km s‑1. This jitter floor likely contains contributions from stellar jitter, instrumental jitter, and undetected companions. We study the impact of the average magnetic field and the distributions of magnetic filling factors on the RV jitter. We find a series of stars with excess RV jitter and distinctive distributions of magnetic filling factors. These stars are characterized by a dominant magnetic field component between 2 to 4 kG. Conclusions. An RV jitter floor can be distinguished from RV jitter induced by activity and rotation based on the stellar equatorial rotation velocity. RV jitter induced by activity and rotation primarily depends on the equatorial rotation velocity. This RV jitter is also related to the distribution of magnetic filling factors, and this emphasizes the role of the magnetic field in the generation of RV jitter.