  Atomic defects in the solid state are a key component of quantum repeater
networks for long-distance quantum communication. Recently, there has been
significant interest in rare earth ions, in particular Er$^{3+}$ for its
telecom-band optical transition, but their application has been hampered by
optical spectral diffusion precluding indistinguishable single photon
generation. In this work we implant Er$^{3+}$ into CaWO$_4$, a material that
combines a non-polar site symmetry, low decoherence from nuclear spins, and is
free of background rare earth ions, to realize significantly reduced optical
spectral diffusion. For shallow implanted ions coupled to nanophotonic cavities
with large Purcell factor, we observe single-scan optical linewidths of 150 kHz
and long-term spectral diffusion of 63 kHz, both close to the Purcell-enhanced
radiative linewidth of 21 kHz. This enables the observation of Hong-Ou-Mandel
interference between successively emitted photons with high visibility,
measured after a 36 km delay line. We also observe spin relaxation times $T_1$
= 3.7 s and $T_2$ > 200 $\mu$s, with the latter limited by paramagnetic
impurities in the crystal instead of nuclear spins. This represents a
significant step towards the construction of telecom-band quantum repeater
networks with single Er$^{3+}$ ions.
