  Low-temperature dynamics of insulating glasses is dominated by a macroscopic
concentration of tunneling two-level systems (TTLS). The distribution of the
switching/relaxation rates of TTLS is exponentially broad, which results in
non-equilibrium state of the glass at arbitrarily long time-scales. Due to the
electric dipolar nature, the switching TTLS generate fluctuating
electromagnetic fields. We study the effect of the non-thermal slow fluctuators
on the dephasing of a solid state qubit. We find that at low enough
temperatures, non-stationary contribution can dominate the stationary (thermal)
one, and discuss how this effect can be minimized.
