  The existence of old neutron stars deeply constrains self-interacting fermion
dark matter, which can form star-killing black holes. We quantify this
constraint on dark matter-nucleon scattering, considering collapse scenarios
that broaden bounds over intermediate masses. We then find the self- and
co-annihilation rates necessary to lift these dark matter-nucleon scattering
bounds. For Yukawa-coupled dark matter that fits dwarf galaxy halo profiles
with a coupling $\alpha = 10^{-1}-10^{-4}$, a scalar mediator mass $m_\phi =
1-500$ MeV, and DM mass $m_X = 0.1-10^7$ GeV, we show that fermion dark matter
is unconstrained if it self-annihilates at a rate greater than $10^{-40} ~
\rm{cm^3/s}$ or co-annihilates with baryons at a rate greater than $10^{-50} ~
\rm{cm^3/s}$.
