  Intermediate Mass Black Holes (IMBHs; 10^1.3-5 M_sun) are thought to form as
relics of Population III stars or from the runaway collapse of stars in young
clusters; their number and very existence are uncertain. We ran N-body
simulations of Galactic IMBHs, modelling them as a halo population distributed
according to a Navarro, Frenk & White (NFW) or a more concentrated Diemand,
Madau & Moore (DMM) density profile. As IMBHs pass through Galactic
molecular/atomic hydrogen regions, they accrete gas, thus becoming X-ray
sources. We constrain the density of Galactic IMBHs, Omega_IMBH, by comparing
the distribution of simulated X-ray sources with the observed one. From the
null detections of Milky Way Ultra-Luminous X-ray sources, and from a
comparison of simulations with unidentified sources in the IBIS/ISGRI catalogue
we find a strong upper limit Omega_IMBH <= 10^-2 Omega_b (<= 10^-1 Omega_b) for
a DMM (NFW) profile, if IMBHs accrete via ADAF disks. Slightly stronger
constraints (Omega_IMBH <= 10^-3 Omega_b for a DMM profile; Omega_IMBH <= 10^-2
Omega_b for a NFW profile) can be derived if IMBHs accrete with higher
efficiency, such as by forming thin accretion disks. Although not very tight,
such constraints are the most stringent ones derived so far in the literature.
