  Despite considerable progress during the last decades in devising a
semiclassical theory for classically chaotic quantum systems a quantitative
semiclassical understanding of their dynamics at late times (beyond the
so-called Heisenberg time $T_H$) is still missing. This challenge,
corresponding to resolving spectral structures on energy scales below the mean
level spacing, is intimately related to the quest for semiclassically restoring
quantum unitarity, which is reflected in real-valued spectral determinants.
Guided through insights for quantum graphs we devise a periodic-orbit
resummation procedure for quantum chaotic systems invoking periodic-orbit self
encounters as the structuring element of a hierarchical phase space dynamics.
We propose a way to purely semiclassically construct real spectral determinants
based on two major underlying mechanisms: (i) Complementary contributions to
the spectral determinant from regrouped pseudo orbits of duration $T < T_H$ and
$T_H-T$ are complex conjugate to each other. (ii) Contributions from long
periodic orbits involving multiple traversals along shorter orbits cancel out.
We furthermore discuss implications for interacting $N$-particle quantum
systems with a chaotic classical large-$N$ limit that have recently attracted
interest in the context of many-body quantum chaos.
