  The complex Fourier transform of the two-point correlator of the energy
spectrum of a quantum system is known as the spectral form factor (SFF). It
constitutes an essential diagnostic tool for phases of matter and quantum
chaos. In black hole physics, it describes the survival probability (fidelity)
of a thermofield double state under unitary time evolution. However, detailed
properties of the SFF of isolated quantum systems with generic spectra are
smeared out by large temporal fluctuations, whose minimization requires
disorder or time averages. This requirement holds for any system size, that is,
the SFF is non-self averaging. Exploiting the fidelity-based interpretation of
this quantity, we prove that using filters, disorder and time averages of the
SFF involve unitarity breaking, i.e., open quantum dynamics described by a
quantum channel that suppresses quantum noise. Specifically, averaging over
Hamiltonian ensembles, time averaging, and frequency filters can be described
by the class of mixed-unitary quantum channels in which information loss can be
recovered. Frequency filters are associated with a time-continuous master
equation generalizing energy dephasing. We also discuss the use of eigenvalue
filters. They are linked to non-Hermitian Hamiltonian evolution without quantum
jumps, whose long-time behavior is described by a Hamiltonian deformation. We
show that frequency and energy filters make the SFF self-averaging at long
times.
