We use the Spitzer Wide-area InfraRed Extragalactic Legacy Survey (SWIRE) to explore the specific star formation activity of galaxies and their evolution near the peak of the cosmic far-infrared (FIR) background at 70 and 160 μm. We use a stacking analysis to determine the mean FIR properties of well-defined subsets of galaxies at flux levels well below the FIR catalogue detection limits of SWIRE and other Spitzer surveys. We tabulate the contribution of different subsets of galaxies to the FIR background at 70 and 160 μm. These long wavelengths provide a good constraint on the bolometric obscured emission. The large area provides good constraints at low z and in finer redshift bins than previous work. At all redshifts we find that the specific FIR luminosity decreases with increasing mass, following a trend LFIR/M* ~ Mβ* with β = -0.38 +/- 0.14. This is a more continuous change than expected from the De Lucia & Blaizot semi-analytic model suggesting modifications to the feedback prescriptions. We see an increase in the specific FIR luminosity by about a factor of ~100 from 0 < z < 2 and find that the specific FIR luminosity evolves as (1 + z)α with α = 4.4 +/- 0.3 for galaxies with 10.5 < log10 M*/Msolar <= 12. This is considerably steeper than the De Lucia & Blaizot semi-analytic model (α ~ 2.5). When separating galaxies into early and late types on the basis of the optical/IR spectral energy distributions we find that the decrease in specific FIR luminosity with stellar mass is stronger in early-type galaxies (β ~ -0.46), while late-type galaxies exhibit a flatter trend (β ~ -0.15). The evolution is strong for both classes but stronger for the early-type galaxies. The early types show a trend of decreasing strength of evolution as we move from lower to higher masses while the evolution of the late-type galaxies has little dependence on stellar mass. We suggest that in late-type galaxies we are seeing a consistently declining specific star formation rate α = 3.36 +/- 0.16 through a common phenomenon, for example, exhaustion of gas supply, i.e. not systematically dependent on the local properties of the galaxy.