  High-Mass X-ray Binaries (HMXBs) are produced after the first supernova event
in a massive binary. These objects are intrinsically young, and can suffer from
a significant natal kick. As such, the progenitors of HMXBs are likely to have
formed away from the current location of the X-ray emitting systems. We aim to
find the birthplace of the known HMXBs of our Milky Way. Specifically, we want
to answer the question whether the formation of HMXBs can be associated to open
stellar clusters and/or Galactic spiral structures, and infer from that the
time elapsed since the first supernova event. We use astrometric data from the
Gaia EDR3 to initialize the position and velocity of each known HMXBs from the
Galaxy, and integrate their motion back in time. In parallel, we perform the
same calculations on a sample of 1381 open clusters detected by Gaia as well as
for four Galactic spiral arms which shape and motion have also been recently
modelled using Gaia data. We report on all the encounter candidates between
HMXBs and clusters or spiral arms in the past 100 Myr. In our sample of 26
HMXBs, we infer that 7 were born in clusters, 8 were born near a Galactic
spiral arm, and conclude that 7 others could have formed isolated. The
birthplaces of the remaining 4 HMXBs are still inconclusive due to a
combination of great distance, poor astrometric data and lack of known open
cluster in the vicinity. We provide the kinematical age since supernova of 15
HMXBs. The astrometry from Gaia and the orbit integration we employ are
effective at finding the birthplaces of HMXBs in the Milky Way. By considering
the biases in our data and method, we find it is likely that the progenitors of
HMXBs preferentially formed alongside other massive stars in open clusters.
