  Neutron stars (NSs) can be used to constrain dark matter (DM) since a NS can
transform into a black hole (BH) if it captures sufficient DM particles and
exceeds the Chandrasekhar limit. We extend earlier work and for the first time
take into account the Galactic motion of individual NSs, which changes the
amount of the captured DM by as large as one to two orders of magnitude. We
systematically apply the analysis to 413 NSs in the Milky Way, and constrain
the DM particle mass and its interaction with nucleon simultaneously. We find
that the most stringent bound is placed by a few NSs and the bound becomes
stronger after considering the Galactic motion. The survival of observed NSs
already excludes a cross section $\sigma_{nX}\gtrsim 10^{-45} \, {\rm cm}^2$
for DM particles with mass from $100\, {\rm MeV}$ to $10^3 \, {\rm GeV}$.
Especially for a mass around $10 \, {\rm GeV}$, the constraint on the cross
section is as stringent as $\sigma_{nX}\lesssim 10^{-49} \, {\rm cm}^2$.
