  In a $p$-spin interaction spherical spin-glass model both the spins and the
couplings are allowed to change in the course of time. The spins are coupled to
a heat bath with temperature $T$, while the coupling constants are coupled to a
bath having temperature $T_{J}$. In an adiabatic limit (where relaxation time
of the couplings is much larger that of the spins) we construct a generalized
two-temperature thermodynamics. It involves entropies of the spins and the
coupling constants. The application for spin-glass systems leads to a standard
replica theory with a non-vanishing number of replicas, $n=T/T_J$. For $p>2$
there occur at low temperatures two different glassy phases, depending on the
value of $n$. The obtained first-order transitions have positive latent heat,
and positive discontinuity of the total entropy. This is the essentially
non-equilibrium effect. The predictions of longtime dynamics and infinite-time
statics differ only for $n<1$ and $p>2$. For $p=2$ correlation of the disorder
(leading to a non-zero $n$) removes the known marginal stability of the spin
glass phase. If the observation time is very large there occurs no
finite-temperature spin glass phase. In this case there are analogies with the
broken-ergodicity dynamics in the usual spin-glass models and non-equilibrium
(aging) dynamics. A generalized fluctuation-dissipation relation is derived.
