  Context. Time delay lensing is a powerful tool to measure the Hubble constant
$H_0$. In order to obtain an accurate estimate of $H_0$ from a sample of time
delay strong lenses, however, it is necessary to have a very good knowledge of
the mass structure of the lens galaxies. Strong lensing data on their own are
not sufficient to break the degeneracy between $H_0$ and the lens model
parameters, on a single object basis.
  Aims. The goal of this study is to determine whether it is possible to break
the $H_0$-lens structure degeneracy with the statistical combination of a large
sample of time-delay lenses, relying purely on strong lensing data (that is,
with no stellar kinematics information).
  Methods. I simulated a set of 100 lenses with doubly imaged quasars and
related time delay measurements. I fitted these data with a Bayesian
hierarchical method and a flexible model for the lens population, emulating the
lens modelling step.
  Results. The sample of 100 lenses, on its own, provides a measurement of
$H_0$ with $3\%$ precision, but with a $-4\%$ bias. However, the addition of
prior information on the lens structural parameters from a large sample of
lenses with no time delays, such as that considered in Paper I, allows for a
$1\%$-level inference. Moreover, the 100 lenses allow for a $0.03$~dex
calibration of galaxy stellar masses, regardless of the level of prior
knowledge of the Hubble constant.
  Conclusions. Breaking the $H_0$-lens model degeneracy with lensing data alone
is possible, but $1\%$ measurements of $H_0$ require either a number of time
delay lenses much larger than 100, or the knowledge of the structural parameter
distribution of the lens population from a separate sample of lenses.
