We present new high spatial resolution (lsim0farcs1) 1-5 μm adaptive optics images, interferometric 1.3 mm continuum and 12CO 2-1 maps, and 350 μm, 2.8 and 3.3 mm fluxes measurements of the HV Tau system. Our adaptive optics images unambiguously demonstrate that HV Tau AB-C is a common proper motion pair. They further reveal an unusually slow orbital motion within the tight HV Tau AB pair that suggests a highly eccentric orbit and/or a large deprojected physical separation. Scattered light images of the HV Tau C edge-on protoplanetary disk suggest that the anisotropy of the dust scattering phase function is almost independent of wavelength from 0.8 to 5 μm, whereas the dust opacity decreases significantly over the same range. The images further reveal a marked lateral asymmetry in the disk that does not vary over a timescale of two years. We further detect a radial velocity gradient in the disk in our 12CO map that lies along the same position angle as the elongation of the continuum emission, which is consistent with Keplerian rotation around a 0.5-1 M sun central star, suggesting that it could be the most massive component in the triple system. To obtain a global representation of the HV Tau C disk, we search for a model that self-consistently reproduces observations of the disk from the visible regime up to millimeter wavelengths. We use a powerful radiative transfer model to compute synthetic disk observations and use a Bayesian inference method to extract constraints on the disk properties. Each individual image, as well as the spectral energy distribution, of HV Tau C can be well reproduced by our models with fully mixed dust provided grain growth has already produced larger-than-interstellar dust grains. However, no single model can satisfactorily simultaneously account for all observations. We suggest that future attempts to model this source include more complex dust properties and possibly vertical stratification. While both grain growth and stratification have already been suggested in many disks, only a panchromatic analysis, such as presented here, can provide a complete picture of the structure of a disk, a necessary step toward quantitatively testing the predictions of numerical models of disk evolution. Data presented in this study were obtained during the course of ESO program 70.C-0565 and IRAM program O048.