  Coherent generation of indistinguishable single photons is crucial for many
quantum communication and processing protocols. Solid-state realizations of
two-level atomic transitions or three-level spin-$\Lambda$ systems offer
significant advantages over their atomic counterparts for this purpose, albeit
decoherence can arise due to environmental couplings. One popular approach to
mitigate dephasing is to operate in the weak excitation limit, where excited
state population is minimal and coherently scattered photons dominate over
incoherent emission. Here we probe the coherence of photons produced using
two-level and spin-$\Lambda$ solid-state systems. We observe that the coupling
of the atomic-like transitions to the vibronic transitions of the crystal
lattice is independent of driving strength and detuning. We apply a polaron
master equation to capture the non-Markovian dynamics of the ground state
vibrational manifolds. These results provide insight into the fundamental
limitations for photon coherence from solid-state quantum emitters, with the
consequence that deterministic single-shot quantum protocols are impossible and
inherently probabilistic approaches must be embraced.
