  We propose policy gradient algorithms for solving a risk-sensitive
reinforcement learning (RL) problem in on-policy as well as off-policy
settings. We consider episodic Markov decision processes, and model the risk
using the broad class of smooth risk measures of the cumulative discounted
reward. We propose two template policy gradient algorithms that optimize a
smooth risk measure in on-policy and off-policy RL settings, respectively. We
derive non-asymptotic bounds that quantify the rate of convergence of our
proposed algorithms to a stationary point of the smooth risk measure. As
special cases, we establish that our algorithms apply to optimization of
mean-variance and distortion risk measures, respectively.
