The polarization analysis of light is typically carried out using modulation schemes. The light of an unknown polarization state is passed through a set of known modulation optics, and a detector is used to measure the total intensity passing the system. The modulation optics is modified several times, and, with the aid of several such measurements, the unknown polarization state of the light can be inferred. How to find the optimal demodulation process has been investigated in the past. However, since the modulation matrix has to be measured for a given instrument and the optical elements can present problems of repeatability, some uncertainty is present in the elements of the modulation matrix or covariances between these elements. We analyze in detail this issue, presenting analytical formulas for calculating the covariance matrix produced by the propagation of such uncertainties on the demodulation matrix, on the inferred Stokes parameters, and on the efficiency of the modulation process. We demonstrate that even if the covariance matrix of the modulation matrix is diagonal, the covariance matrix of the demodulation matrix is in general nondiagonal because matrix inversion is a nonlinear operation. This propagates through the demodulation process and induces correlations on the inferred Stokes parameters.