In this paper of ERGO-ML (Extracting Reality from Galaxy Observables with Machine Learning), we develop a model that infers the merger/assembly histories of galaxies directly from optical images. We apply the self-supervised contrastive learning framework Nearest Neighbour Contrastive Learning (NNCLR) on realistic HSC mock images (g, r, i bands) produced from galaxies simulated within the TNG50 and TNG100 runs of the IllustrisTNG project and with stellar masses of $10^{9-12} \rm {M}_{\odot }$. The resulting representation is then used as conditional input for a conditional Invertible Neural Network (cINN) to gain posteriors for merger/assembly statistics, particularly the lookback time and stellar mass of the last major merger and the fraction of ex situ stars. We achieve good accuracy in inferring the stellar ex situ fraction ($\le \pm$10 per cent for 80 per cent of the test sample). The information content about the lookback time is, instead, limited. We also successfully apply the TNG-trained model to simulated mocks from the EAGLE simulation, demonstrating that our model is applicable outside of the TNG domain. We hence use our simulation-based model to infer aspects of the history of observed galaxies, in particular for HSC images that are close to the domain of TNG ones. We recover the trend of increasing ex situ stellar fraction with stellar mass and more spherical morphology, but we also identify a discrepancy between TNG and HSC: on average, observed galaxies generally exhibit lower ex situ fractions. Despite challenges such as information loss and domain shifts, our results demonstrate the feasibility of extracting the merger past of galaxies from their optical images.