Aims:We study the structure and reveal the physical nature of the reversed granulation pattern in the solar photosphere by means of 3-dimensional radiative hydrodynamics simulations. Methods: We used the MURaM code to obtain a realistic model of the near-surface layers of the convection zone and the photosphere. Results: The pattern of horizontal temperature fluctuations at the base of the photosphere consists of relatively hot granular cells bounded by the cooler intergranular downflow network. With increasing height in the photosphere, the amplitude of the temperature fluctuations diminishes. At a height of z=130-140 km in the photosphere, the pattern of horizontal temperature fluctuations reverses so that granular regions become relatively cool compared to the intergranular network. Detailed analysis of the trajectories of fluid elements through the photosphere reveal that the motion of the fluid is non-adiabatic, owing to strong radiative cooling when approaching the surface of optical depth unity followed by reheating by the radiation field from below. The temperature structure of the photosphere results from the competition between expansion of rising fluid elements and radiative heating. The former acts to lower the temperature of the fluid whereas the latter acts to increase it towards the radiative equilibrium temperature with a net entropy gain. After the fluid overturns and descends towards the convection zone, radiative energy loss again decreases the entropy of the fluid. Radiative heating and cooling of fluid elements that penetrate into the photosphere and overturn do not occur in equal amounts. The imbalance in the cumulative heating and cooling of these fluid elements is responsible for the reversal of temperature fluctuations with respect to height in the photosphere.