  Off-policy learning is powerful for reinforcement learning. However, the high
variance of off-policy evaluation is a critical challenge, which causes
off-policy learning falls into an uncontrolled instability. In this paper, for
reducing the variance, we introduce control variate technique to
$\mathtt{Expected}$ $\mathtt{Sarsa}$($\lambda$) and propose a tabular
$\mathtt{ES}$($\lambda$)-$\mathtt{CV}$ algorithm. We prove that if a proper
estimator of value function reaches, the proposed
$\mathtt{ES}$($\lambda$)-$\mathtt{CV}$ enjoys a lower variance than
$\mathtt{Expected}$ $\mathtt{Sarsa}$($\lambda$). Furthermore, to extend
$\mathtt{ES}$($\lambda$)-$\mathtt{CV}$ to be a convergent algorithm with linear
function approximation, we propose the $\mathtt{GES}$($\lambda$) algorithm
under the convex-concave saddle-point formulation. We prove that the
convergence rate of $\mathtt{GES}$($\lambda$) achieves $\mathcal{O}(1/T)$,
which matches or outperforms lots of state-of-art gradient-based algorithms,
but we use a more relaxed condition. Numerical experiments show that the
proposed algorithm performs better with lower variance than several
state-of-art gradient-based TD learning algorithms: $\mathtt{GQ}$($\lambda$),
$\mathtt{GTB}$($\lambda$) and $\mathtt{ABQ}$($\zeta$).
