We track small magnetic structures on very quiet regions (internetwork) of the Sun. We follow the footpoints of small-scale magnetic loops that appear on the photosphere at granular scales using spectropolarimetric and magnetographic data obtained with Hinode. We find two different regimes for their wanderings. Within granules (where they appear), they seem to be passively advected by the plasma - which is justified by their relatively low magnetic flux (~1016 Mx), and magnetic field strength (~200 G). The plasma flow thus traced is roughly laminar with a characteristic mean velocity of 2 km s-1 and very low vorticity. Once the magnetic markers reach intergranular lanes, they remain there and are buffeted by the random flows of neighbouring granules and turbulent intergranules, follow random walks, and disperse across the solar surface with a diffusion constant of 195 km2 s-1. While on their intergranular random walking, they may fall close to whirlpools (on scales ≲400 km) associated with convective downdrafts, similar to the events recently reported in mesogranular and supergranular cell boundaries tracking magnetic bright points, which provides additional evidence that these events are ubiquitous on the solar surface.