  The energetics, spectrum, and composition of cosmic rays with energies below
about $10^{15}$ eV are fairly well explained by models involving supernova
shocks. In contrast, no widely accepted theory exists for the origin of
ultra-high energy cosmic rays (UHECRs), which have energies above $10^{15}$ eV.
Instead of proposing a specific model, here we place strong constraints on any
model of UHECRs involving isolated neutron stars (no companions). We consider
the total power requirements and show that the only viable power source
associated with isolated neutron stars is rotation. Mechanisms based on
accretion from the interstellar medium fall short of the necessary power
despite the most optimistic assumptions. Power considerations also demonstrate
that not enough rotational energy is tapped by a "propeller"-like acceleration
of interstellar matter. The most promising source of energy is rotational
spindown via magnetic braking. We examine microphysical energy loss processes
near magnetized neutron stars and conclude that the most likely site for
yielding UHECRs from isolated neutron stars is near or beyond the light
cylinder.
