  We investigate the frictional force arising from quantum fluctuations when
two dissipative metallic plates are set in a shear motion. While early studies
showed that the electromagnetic fields in the quantum friction setup reach
nonequilibrium steady states, yielding a time-independent force, other works
have demonstrated the failure to attain steady states, leading to instability
and time-varying friction under sufficiently low-loss conditions. Here, we
develop a fully quantum-mechanical theory without perturbative approximations
and unveil the transition from stable to unstable regimes of the quantum
friction setup. Due to the relative motion of the plates, their electromagnetic
response may be active in some conditions, resulting in optical gain. We prove
that the standard fluctuation-dissipation leads to inconsistent results when
applied to our system, and, in particular, it predicts a vanishing frictional
force. Using a modified fluctuation-dissipation relation tailored for gain
media, we calculate the frictional force in terms of the system Green's
function, thereby recovering early works on quantum friction. Remarkably, we
also find that the frictional force diverges to infinity as the relative
velocity of the plates approaches a threshold. This threshold is determined by
the damping strength and the distance between the metal surfaces. Beyond this
critical velocity, the system exhibits instability, akin to the behaviour of a
laser cavity, where no steady state exists. In such a scenario, the frictional
force escalates exponentially. Our findings pave the way for experimental
exploration of the frictional force in proximity to this critical regime.
