We present high time-resolution multicolour observations of the quiescent soft X-ray transient XTE J1118+480 obtained with ULTRACAM. Superimposed on the double-humped continuum g'- and i'-band light curves are rapid flare events which typically last a few minutes. The power density spectrum of the light curves can be described by a broken power-law model with a break frequency at ~2 mHz or a power-law model plus a broad quasi-periodic oscillation at ~2 mHz. In the context of the cellular automaton we estimate the size of the quiescent advection-dominated accretion flow (ADAF) region to be ~104 Schwarzschild radii, similar to that observed in other quiescent black hole X-ray transients, suggesting the same underlying physics. The similarities between the low/hard and quiescent-state power density spectrum suggest a similar origin for the optical and X-ray variability, most likely from regions at or near the ADAF.