We present measurements of the auto- and cross-frequency power spectra of the cosmic infrared background (CIB) at 250, 350, and 500 μm (1200, 860, and 600 GHz) from observations totaling ~70 deg2 made with the SPIRE instrument aboard the Herschel Space Observatory. We measure a fractional anisotropy δI/I = 14% ± 4%, detecting signatures arising from the clustering of dusty star-forming galaxies in both the linear (2-halo) and nonlinear (1-halo) regimes; and that the transition from the 2- to 1-halo terms, below which power originates predominantly from multiple galaxies within dark matter halos, occurs at k θ ~ 0.10-0.12 arcmin–1 (l ~ 2160-2380), from 250 to 500 μm. New to this paper is clear evidence of a dependence of the Poisson and 1-halo power on the flux-cut level of masked sources—suggesting that some fraction of the more luminous sources occupy more massive halos as satellites, or are possibly close pairs. We measure the cross-correlation power spectra between bands, finding that bands which are farthest apart are the least correlated, as well as hints of a reduction in the correlation between bands when resolved sources are more aggressively masked. In the second part of the paper, we attempt to interpret the measurements in the framework of the halo model. With the aim of fitting simultaneously with one model the power spectra, number counts, and absolute CIB level in all bands, we find that this is achievable by invoking a luminosity-mass relationship, such that the luminosity-to-mass ratio peaks at a particular halo mass scale and declines toward lower and higher mass halos. Our best-fit model finds that the halo mass which is most efficient at hosting star formation in the redshift range of peak star-forming activity, z ~ 1-3, is log(M peak/M ☉) ~ 12.1 ± 0.5, and that the minimum halo mass to host infrared galaxies is log(M min/M ☉) ~ 10.1 ± 0.6. Herschel is an ESA space observatory with science instruments provided by European-led Principal Investigator consortia and with important participation from NASA.