Early-type dwarfs (dEs) are by far the most abundant galaxy population in nearby clusters. Whether these objects are primordial, or the recent end products of the different physical mechanisms that can transform galaxies once they enter these high-density environments, is still a matter of debate. Here we present a novel approach to test these scenarios by comparing the properties of the globular cluster systems (GCSs) of Virgo dEs and their potential progenitors with simple predictions from gravitational and hydrodynamical interaction models. We show that low-mass (M★ ≲ 2 × 108 M&sun;) dEs have GCSs consistent with the descendants of gas-stripped late-type dwarfs. On the other hand, higher mass dEs have properties - including the high mass specific frequencies of their GCSs and their concentrated spatial distribution within Virgo - incompatible with a recent, environmentally driven evolution. They mostly comprise nucleated systems, but also dEs with recent star formation and/or disc features. Bright, nucleated dEs appear to be a population that has long resided within the cluster potential well, but have surprisingly managed to retain very rich and spatially extended GCSs - possibly an indication of high total masses. Our analysis does not favour violent evolutionary mechanisms that result in significant stellar mass-losses, but more gentle processes involving gas removal by a combination of internal and external factors, and highlights the relevant role of initial conditions. Additionally, we briefly comment on the origin of luminous cluster S0 galaxies.