  Photon pairs produced by parametric down-conversion or four-wave mixing can
interfere with each other in multiport interferometers, or carry entanglement
between distant nodes for use in entanglement swapping. This requires the
photons be spectrally pure to ensure good interference, and have high heralding
efficiency to know accurately the number of photons involved and to maintain
high rates as the number of photons grows. Spectral filtering is often used to
remove noise and define spectral properties. For heralded single photons high
purity and heralding efficiency is possible by filtering the heralding arm, but
when both photons in typical pair sources are filtered, we show that the
heralding efficiency of one or both of the photons is strongly reduced even by
ideal spectral filters with 100% transmission in the passband: any improvement
in reduced-state spectral purity from filtering comes at the cost of lowered
heralding efficiency. We consider the fidelity to a pure, lossless single
photon, symmetrize it to include both photons of the pair, and show this
quantity is intrinsically limited for sources with spectral correlation. We
then provide a framework for this effect for benchmarking common photon pair
sources, and present an experiment where we vary the photon filter bandwidths
and measure the increase in purity and corresponding reduction in heralding
efficiency.
