We conduct a comprehensive numerical study of the orbital dependence of harassment on early-type dwarfs consisting of 168 different orbits within a realistic, Virgo-like cluster, varying in eccentricity and pericentre distance. We find harassment is only effective at stripping stars or truncating their stellar discs for orbits that enter deep into the cluster core. Comparing to the orbital distribution in cosmological simulations, we find that the majority of the orbits (more than three quarters) result in no stellar mass loss. We also study the effects on the radial profiles of the globular cluster systems of early-type dwarfs. We find these are significantly altered only if harassment is very strong. This suggests that perhaps most early-type dwarfs in clusters such as Virgo have not suffered any tidal stripping of stars or globular clusters due to harassment, as these components are safely embedded deep within their dark matter halo. We demonstrate that this result is actually consistent with an earlier study of harassment of dwarf galaxies, despite the apparent contradiction. Those few dwarf models that do suffer stellar stripping are found out to the virial radius of the cluster at redshift = 0, which mixes them in with less strongly harassed galaxies. However when placed on phase-space diagrams, strongly harassed galaxies are found offset to lower velocities compared to weakly harassed galaxies. This remains true in a cosmological simulation, even when haloes have a wide range of masses and concentrations. Thus phase-space diagrams may be a useful tool for determining the relative likelihood that galaxies have been strongly or weakly harassed.