  We propose a simple test for the existence of a cluster of black hole
remnants around Sgr A* that is based on a small sample of any type of Galactic
Center objects, provided they are substantially less massive than the black
holes and constitute part of an old (> 1 Gyr) population. The test relies on
the fact that, under the presence of such a cluster of heavy remnants and
because of energy equipartition, lower mass objects would be expelled from the
central regions and settle into a distribution very different than the cusp
expected to be induced by the supermassive black hole alone. We show that with
a sample of just 50 objects and using only their angular positions on the sky
relative to Sgr A* it is possible to clearly differentiate between a
distribution consistent with the presence of the cluster of black holes and a
power-law cusp distribution. We argue that millisecond pulsars might currently
be the best candidate to perform this test, because of the large uncertainties
involved in the age determination of less exotic objects. In addition, by
measuring their first and second period derivatives, millisecond pulsars offer
the rare opportunity of determining the complete phase space information of the
objects. We show that this extra information improves the detection of mass
segregation by about 30%.
