We investigate the fine structure of magnetic fields in the atmosphere of the quiet Sun. We use photospheric magnetic field measurements from SUNRISE/IMaX with unprecedented spatial resolution to extrapolate the photospheric magnetic field into higher layers of the solar atmosphere with the help of potential and force-free extrapolation techniques. We find that most magnetic loops that reach into the chromosphere or higher have one footpoint in relatively strong magnetic field regions in the photosphere. Ninety-one percent of the magnetic energy in the mid-chromosphere (at a height of 1 Mm) is in field lines, whose stronger footpoint has a strength of more than 300 G, i.e., above the equipartition field strength with convection. The loops reaching into the chromosphere and corona are also found to be asymmetric in the sense that the weaker footpoint has a strength B < 300 G and is located in the internetwork (IN). Such loops are expected to be strongly dynamic and have short lifetimes, as dictated by the properties of the IN fields.