  Motivated by risk-sensitive reinforcement learning scenarios, we consider the
problem of policy evaluation for variance in a discounted reward Markov
decision process (MDP). For this problem, a temporal difference (TD) type
learning algorithm with linear function approximation (LFA) exists in the
literature, though only asymptotic guarantees are available for this algorithm.
We derive finite sample bounds that hold (i) in the mean-squared sense; and
(ii) with high probability, when tail iterate averaging is employed
with/without regularization. Our bounds exhibit exponential decay for the
initial error, while the overall bound is $O(1/t)$, where $t$ is the number of
update iterations of the TD algorithm. Further, the bound for the regularized
TD variant is for a universal step size. Our bounds open avenues for analysis
of actor-critic algorithms for mean-variance optimization in a discounted MDP.
