We present results of an extensive study of the X-ray spectral properties of sources detected in the RIXOS survey, which is a large, nearly complete sample of objects detected serendipitously in ROSAT PSPC fields down to a flux limit of 3x10^-14ergcm^-2s^-1 (0.5-2keV). We show that for X-ray surveys containing sources with low count rate, such as RIXOS, spectral slopes estimated using simple hardness ratios in the ROSAT band can be biased. Instead, we analyse three-colour X-ray data using statistical techniques appropriate to the Poisson regime which remove the effects of this bias. We also show that the use of three-colour data enables some discrimination between thermal and non-thermal spectra. We have then applied this technique to the RIXOS survey to study the spectral properties of the sample. For the AGN we find an average energy index of 1.05+/-0.05, with no evidence for spectral evolution with redshift. Individual AGN are shown to have a range of properties, including soft X-ray excesses and intrinsic absorption. Narrow-emission-line galaxies (NELGs) also seem to fit to a power-law spectrum, which may indicate a non-thermal origin for their X-ray emission. We infer that most of the clusters in the sample have a bremsstrahlung temperature >3keV, although some show evidence for a cooling flow. The stars deviate strongly from a power-law model but fit to a thermal model. Finally, we have analysed the whole RIXOS sample (extending the flux cut-off to the sensitivity threshold of each individual observation) containing 1762 sources to study the relationship between spectral slope and flux. We find that the mean spectral slope of the sources hardens at lower fluxes, in agreement with results from other samples. However, a study of the individual sources demonstrates that the majority have relatively soft spectra even at faint flux levels, and the hardening of the mean is caused by the appearance of a population of very hard sources at the lowest fluxes. This has implications for the nature of the soft X-ray background.