  We study the long term dynamical evolution of stellar mass black holes (BHs)
at the Galactic center (GC) and put constraints on their number and central
mass distribution. Models of the GC are considered that have not yet achieved a
steady state under the influence of random gravitational encounters. Contrary
to some recent claims that mass-segregation can rapidly rebuild a density cusp
in the stars, we find that time scales associated with cusp regrowth are longer
than the Hubble time. These results cast doubts on standard models that
postulate high densities of BHs near the GC and motivate studies that start
from initial conditions which correspond to well-defined physical models. For
the first time, we consider the distribution of BHs in a dissipationless
formation model for the Milky Way nuclear cluster (NC), in which massive
stellar clusters merge in the GC to form a nucleus. We simulate the successive
inspiral of massive clusters containing an inner dense cluster of BHs. The
pre-existing mass segregation is not completely erased as the clusters are
disrupted by the massive black hole tidal field. As a result, after 12 inspiral
events a NC forms in which the BHs have higher central densities than the
stars. After evolving the model for 5-10 Gyr, the BHs do form a steep central
cusp, while the stellar distribution maintains properties that resemble those
of the Milky Way NC. Finally, we investigate the effect of BH perturbations on
the motion of the GC S-stars, as a means of constraining the number of the
perturbers. We find that reproducing the S-star orbital distribution requires
>~1000 BHs within 0.1 pc of Sgr A*. A dissipationless formation scenario for
the Milky Way NC is consistent with this lower limit and therefore could
reconcile the need for high central densities of BHs (to explain the orbits of
the S-stars), with the missing-cusp problem of the GC giant star population.
