  If millisecond pulsars (MSPs) are responsible for the excess gamma-ray
emission observed from the region surrounding the Galactic Center, the same
region should also contain a large population of low-mass X-ray binaries
(LMXBs). In this study, we compile and utilize a sizable catalog of LMXBs
observed in the the Milky Way's globular cluster system and in the Inner
Galaxy, as well as the gamma-ray emission observed from globular clusters, to
estimate the flux of gamma rays predicted from MSPs in the Inner Galaxy. From
this comparison, we conclude that only up to $\sim$4-23% of the observed
gamma-ray excess is likely to originate from MSPs. This result is consistent
with, and more robust than, previous estimates which utilized smaller samples
of both globular clusters and LMXBs. If MSPs had been responsible for the
entirety of the observed excess, INTEGRAL should have detected $\sim$$10^3$
LMXBs from within a $10^{\circ}$ radius around the Galactic Center, whereas
only 42 LMXBs (and 46 additional LMXB candidates) have been observed.
