  The dynamical formation of black hole binaries in globular clusters that
merge due to gravitational waves occurs more frequently in higher stellar
density. Meanwhile, the probability to form intermediate mass black holes
(IMBHs) also increases with the density. To explore the impact of the formation
and growth of IMBHs on the population of stellar mass black hole binaries from
globular clusters, we analyze the existing large survey of Monte-Carlo globular
cluster simulation data (MOCCA SURVEY Database I). We show that the number of
binary black hole mergers agrees with the prediction based on clusters' initial
properties when the IMBH mass is not massive enough or the IMBH seed forms at a
later time. However, binary black hole formation and subsequent merger events
are significantly reduced compared to the prediction when the present-day IMBH
mass is more massive than $\sim10^4 \rm M_{\odot}$ or the present-day IMBH mass
exceeds about 1 per cent of cluster's initial total mass. By examining the
maximum black hole mass in the system at the moment of black hole binary
escaping, we find that $\sim$ 90 per cent of the merging binary black holes
escape before the formation and growth of the IMBH. Furthermore, large fraction
of stellar mass black holes are merged into the IMBH or escape as single black
holes from globular clusters in cases of massive IMBHs, which can lead to the
significant under-population of binary black holes merging with gravitational
waves by a factor of 2 depending on the clusters' initial distributions.
