  Understanding and mitigating loss channels due to two-level systems (TLS) is
one of the main cornerstones in the quest of realizing long photon lifetimes in
superconducting quantum circuits. Typically, the TLS to which a circuit couples
are modeled as a large bath without any coherence. Here we demonstrate that the
coherence of TLS has to be considered to accurately describe the ring-down
dynamics of a coaxial quarter-wave resonator with an internal quality factor of
$0.5\times10^9$ at the single-photon level. The transient analysis reveals
effective non-Markovian dynamics of the combined TLS and cavity system, which
we can accurately fit by introducing a comprehensive TLS model. The fit returns
an average coherence time of around $T_2^*\approx0.3\,\mathrm{\mu s}$ for a
total of $N\approx10^{9}$ TLS with power-law distributed coupling strengths.
Despite the shortly coherent TLS excitations, we observe long-term effects on
the cavity decay due to coherent elastic scattering between the resonator field
and the TLS. Moreover, this model provides an accurate prediction of the
internal quality factor's temperature dependence.
