  We study the spectral gap for transfer operators of the skew product $F:
\mathbb{T}^d\times \mathbb{T}^\ell\to \mathbb{T}^d\times \mathbb{T}^\ell$ given
by $F(x,y)=(Tx, y+\tau(x) \pmod{ \mathbb{Z}^\ell})$, where $T: \mathbb{T}^d\to
\mathbb{T}^d$ is a $C^\infty$ uniformly expanding endomorphism, and the fiber
map $\tau: \mathbb{T}^d\to \mathbb{R}^\ell$ is a $C^\infty$ map. We construct a
Hilbert space $\mathcal{W}^{-s}$ for any $s<0$, which contains all the H\"older
functions of H\"older exponents $|s|$ on $ \mathbb{T}^d\times \mathbb{T}^\ell$.
Applying the method of semiclassical analysis, we obtain the dichotomy: either
the transfer operator has a spectral gap on $\mathcal{W}^{-s}$, or $\tau$ is an
essential coboundary. In the former case, $F$ mixes exponentially fast for
H\"older observables with H\"older exponents $|s|$; and in the latter case,
either $F$ is not weak mixing and it is semiconjugate to a circle rotation, or
$F$ is unstably mixing, i.e., it can be approximated by non-mixing skew
products.
