All observations by the aperture photometer (PHT-P) and the far-infrared (FIR) camera section (PHT-C) of ISOPHOT included reference measurements against stable internal fine calibration sources (FCS) to correct for temporal drifts in detector responsivities. The FCSs were absolutely calibrated in-orbit against stars, asteroids and planets, covering wavelengths from 3.2 to 240 mu m. We present the calibration concept for point sources within a flux-range from 60 mJy up to 4500 Jy for staring and raster observations in standard configurations and discuss the requisite measurements and the uncertainties involved. In this process we correct for instrumental effects like nonlinearities, signal transients, time variable dark current, misalignments and diffraction effects. A set of formulae is developed that describes the calibration from signal level to flux densities. The scatter of 10 to 20% of the individual data points around the derived calibration relations is a measure of the consistency and typical accuracy of the calibration. The reproducibility over longer periods of time is better than 10%. The calibration tables and algorithms have been implemented in the final versions of the software for offline processing and interactive analysis.