A fundamental prediction of the ΛCDM cosmology is the hierarchical build-up of structure and therefore the successive merging of galaxies into more massive ones. As one can only observe galaxies at one specific time in the cosmic history, this merger history remains, in principle, unobservable. By using the TNG100 simulation of the IllustrisTNG project, we show that it is possible to infer the unobservable stellar assembly and merger history of central galaxies from their observable properties by using machine learning techniques. In particular, in this first paper of ERGO-ML (Extracting Reality from Galaxy Observables with Machine Learning), we choose a set of seven observable integral properties of galaxies to infer the stellar ex-situ fraction, the average merger lookback times and mass ratios, and the lookback time and stellar mass of the last major merger. To infer the posterior distribution for these parameters and hence estimate the uncertainties in the predictions, we use a conditional Invertible Neural Network (cINN). We find that the stellar ex-situ fraction and the time of the last major merger are well-determined by the selected set of observables, that the mass-weighted merger mass ratio is unconstrained, and that, beyond stellar mass, stellar morphology and stellar age are the most informative properties. Finally, we show that the cINN recovers the remaining unexplained scatter and secondary cross-correlations. Overall, this is a first step towards a tool that can be applied to large galaxy surveys in order to infer unobservable properties of the galaxies' past, enabling empirical studies of galaxy evolution enriched by cosmological simulations.