  Keeping risk under control is often more crucial than maximizing expected
rewards in real-world decision-making situations, such as finance, robotics,
autonomous driving, etc. The most natural choice of risk measures is variance,
which penalizes the upside volatility as much as the downside part. Instead,
the (downside) semivariance, which captures the negative deviation of a random
variable under its mean, is more suitable for risk-averse proposes. This paper
aims at optimizing the mean-semivariance (MSV) criterion in reinforcement
learning w.r.t. steady reward distribution. Since semivariance is
time-inconsistent and does not satisfy the standard Bellman equation, the
traditional dynamic programming methods are inapplicable to MSV problems
directly. To tackle this challenge, we resort to Perturbation Analysis (PA)
theory and establish the performance difference formula for MSV. We reveal that
the MSV problem can be solved by iteratively solving a sequence of RL problems
with a policy-dependent reward function. Further, we propose two on-policy
algorithms based on the policy gradient theory and the trust region method.
Finally, we conduct diverse experiments from simple bandit problems to
continuous control tasks in MuJoCo, which demonstrate the effectiveness of our
proposed methods.
