Variations in the stellar initial mass function (IMF) have been invoked to explain the spectroscopic and dynamical properties of early-type galaxies (ETGs). However, no observations have yet been able to disentangle the physical driver. We analyze here a sample of 24 ETGs drawn from the CALIFA survey, deriving in a homogeneous way their stellar population and kinematic properties. We find that the local IMF is tightly related to the local metallicity, becoming more bottom-heavy toward metal-rich populations. Our result, combined with the galaxy mass–metallicity relation, naturally explains previous claims of a galaxy mass–IMF relation, derived from non-IFU spectra. If we assume that—within the star formation environment of ETGs—metallicity is the main driver of IMF variations, a significant revision of the interpretation of galaxy evolution observables is necessary.