We exploit the deep and extended far-IR data sets (at 70, 100 and 160 μm) of the Herschel Guaranteed Time Observation (GTO) PACS Evolutionary Probe (PEP) Survey, in combination with the Herschel Multi-tiered Extragalactic Survey data at 250, 350 and 500 μm, to derive the evolution of the rest-frame 35-, 60-, 90- and total infrared (IR) luminosity functions (LFs) up to z ˜ 4. We detect very strong luminosity evolution for the total IR LF (LIR ∝ (1 + z)3.55 ± 0.10 up to z ˜ 2, and ∝ (1 + z)1.62 ± 0.51 at 2 < z ≲ 4) combined with a density evolution (∝(1 + z)-0.57 ± 0.22 up to z ˜ 1 and ∝ (1 + z)-3.92 ± 0.34 at 1 < z ≲ 4). In agreement with previous findings, the IR luminosity density (ρIR) increases steeply to z ˜ 1, then flattens between z ˜ 1 and z ˜ 3 to decrease at z ≳ 3. Galaxies with different spectral energy distributions, masses and specific star formation rates (SFRs) evolve in very different ways and this large and deep statistical sample is the first one allowing us to separately study the different evolutionary behaviours of the individual IR populations contributing to ρIR. Galaxies occupying the well-established SFR-stellar mass main sequence (MS) are found to dominate both the total IR LF and ρIR at all redshifts, with the contribution from off-MS sources (≥0.6 dex above MS) being nearly constant (˜20 per cent of the total ρIR) and showing no significant signs of increase with increasing z over the whole 0.8 < z < 2.2 range. Sources with mass in the range 10 ≤ log(M/M⊙) ≤ 11 are found to dominate the total IR LF, with more massive galaxies prevailing at the bright end of the high-z (≳2) LF. A two-fold evolutionary scheme for IR galaxies is envisaged: on the one hand, a starburst-dominated phase in which the Super Massive Black Holes (SMBH) grows and is obscured by dust (possibly triggered by a major merging event), is followed by an AGN-dominated phase, then evolving towards a local elliptical. On the other hand, moderately star-forming galaxies containing a low-luminosity AGN have various properties suggesting they are good candidates for systems in a transition phase preceding the formation of steady spiral galaxies.