Aims: We investigate the relationship between the X-ray variability amplitude and X-ray luminosity for a sample of 14 bright ultra-luminous X-ray sources (ULXs) with XMM-Newton/EPIC data, and compare it with the well-established, similar relationship for active galactic nuclei (AGN). Methods: We computed the normalised excess variance in the 2-10 keV light curves of these objects and their 2-10 keV band intrinsic luminosity L2-10 keV. We also determined model "variability-luminosity" relationships for AGN, under several assumptions regarding their power-spectral shape. We compared these model predictions at low luminosities with the ULX data. Results: The variability amplitude of the ULXs is significantly smaller than expected from a simple extrapolation of the AGN "variability-luminosity" relationship at low luminosities. We also find evidence of an anti-correlation between the variability amplitude and L2-10 keV for ULXs. The shape of this relationship is consistent with the AGN data but only if the ULXs data are shifted by four orders of magnitudes in luminosity. Conclusions: Most (but not all) of the ULXs could be "scaled-down" version of AGN if we assume that i) their black hole mass and accretion rate are between ~(2.5-30)× 103 M&sun; and ~1-80% of the Eddington limit and ii) their power spectral density has a doubly broken power-law shape. This shape and accretion rate is consistent with Galactic black hole systems operating in their so-called "low-hard" and "very-high" states.