  In this paper, we study the Temporal Difference (TD) learning with linear
value function approximation. It is well known that most TD learning algorithms
are unstable with linear function approximation and off-policy learning. Recent
development of Gradient TD (GTD) algorithms has addressed this problem
successfully. However, the success of GTD algorithms requires a set of well
chosen features, which are not always available. When the number of features is
huge, the GTD algorithms might face the problem of overfitting and being
computationally expensive. To cope with this difficulty, regularization
techniques, in particular $\ell_1$ regularization, have attracted significant
attentions in developing TD learning algorithms. The present work combines the
GTD algorithms with $\ell_1$ regularization. We propose a family of $\ell_1$
regularized GTD algorithms, which employ the well known soft thresholding
operator. We investigate convergence properties of the proposed algorithms, and
depict their performance with several numerical experiments.
