  During the inspiral and merger of a binary black hole, gravitational
radiation is emitted anisotropically due to asymmetries in the merger
configuration. This anisotropic radiation leads to a gravitational wave kick,
or recoil velocity, as large as ~ 4000 km/sec. We investigate the effect
gravitational recoil has on the retention of intermediate mass black holes
(IMBH) within Galactic globular clusters. Assuming that our current
understanding of IMBH-formation is correct and yields an IMBH-seed in every
globular cluster, we find a significant problem retaining low mass IMBHs (1000
$\Msun$) in the typical merger-rich globular cluster environment. Given a
uniform black hole spin distribution and orientation and a Kroupa IMF, we find
that at most 3% of the globular clusters can retain an IMBH larger than 1000
$\Msun$ today. For a population of black holes that better approximates mass
loss from winds and supernovae, we find that 16% of globulars can retain an
IMBH larger than 1000 $\Msun$. Our calculations show that if there are black
holes of mass $M > 60 \Msun$ in a cluster, repeated IMBH-BH encounters will
eventually eject a 1000 $\Msun$ IMBH with greater than 30% probability. As a
consequence, a large population of rogue black holes may exist in our Milky Way
halo. We discuss the dynamical implications of this subpopulation, and its
possible connection to ultraluminous X-ray sources (ULXs).
