  The gravitational coupling of nearby massive bodies to test masses in a
gravitational wave (GW) detector cannot be shielded, and gives rise to 'gravity
gradient noise' (GGN) in the detector. In this paper we show that for any GW
detector using local test masses in the Inner Solar System, the GGN from the
motion of the field of $\sim 10^5$ Inner Solar System asteroids presents an
irreducible noise floor for the detection of GW that rises exponentially at low
frequencies. This severely limits prospects for GW detection using local test
masses for frequencies $f_{\text{GW}} \lesssim (\text{few})\times 10^{-7}\,$Hz.
At higher frequencies, we find that the asteroid GGN falls rapidly enough that
detection may be possible; however, the incompleteness of existing asteroid
catalogs with regard to small bodies makes this an open question around
$f_{\text{GW}}\sim \mu$Hz, and further study is warranted. We show that a
detector network placed in the Outer Solar System would not be overwhelmed by
this noise above $\sim 10\,$nHz, and make comments on alternative approaches
that could overcome the limitations of local test masses for GW detection in
the $\sim 10\,$nHz-$\mu$Hz band.
