The role of the magnetic field in the formation of the filamentary structures observed in the interstellar medium (ISM) is a debated topic owing to the paucity of relevant observations needed to test existing models. The Planck all-sky maps of linearly polarized emission from dust at 353 GHz provide the required combination of imaging and statistics to study the correlation between the structures of the Galactic magnetic field and of interstellar matter over the whole sky, both in the diffuse ISM and in molecular clouds. The data reveal that structures, or ridges, in the intensity map have counterparts in the Stokes Q and/or U maps. We focus our study on structures at intermediate and high Galactic latitudes, which cover two orders of magnitude in column density, from 1020 to 1022 cm-2. We measure the magnetic field orientation on the plane ofthe sky from the polarization data, and present an algorithm to estimate the orientation of the ridges from the dust intensity map. We use analytical models to account for projection effects. Comparing polarization angles on and off the structures, we estimate the mean ratio between the strengths of the turbulent and mean components of the magnetic field to be between 0.6 and 1.0, with a preferred value of 0.8. We find that the ridges are usually aligned with the magnetic field measured on the structures. This statistical trend becomes more striking for increasing polarization fraction and decreasing column density. There is no alignment for the highest column density ridges. We interpret the increase in alignment with polarization fraction as a consequence of projection effects. We present maps to show that the decrease in alignment for high column density is not due to a loss of correlation between the distribution of matter and the geometry of the magnetic field. In molecular complexes, we also observe structures perpendicular to the magnetic field, which, statistically, cannot be accounted for by projection effects. This first statistical study of the relative orientation between the matter structures and the magnetic field in the ISM points out that, at the angular scales probed by Planck, the field geometry projected on the plane of the sky is correlated with the distribution of matter. In the diffuse ISM, the structures of matter are usually aligned with the magnetic field, while perpendicular structures appear in molecular clouds. We discuss our results in the context of models and MHD simulations, which attempt to describe the respective roles of turbulence, magnetic field, and self-gravity in the formation of structures in the magnetized ISM.