  Real-world applications require RL algorithms to act safely. During learning
process, it is likely that the agent executes sub-optimal actions that may lead
to unsafe/poor states of the system. Exploration is particularly brittle in
high-dimensional state/action space due to increased number of low-performing
actions. In this work, we consider risk-averse exploration in approximate RL
setting. To ensure safety during learning, we propose the distributionally
robust policy iteration scheme that provides lower bound guarantee on
state-values. Our approach induces a dynamic level of risk to prevent poor
decisions and yet preserves the convergence to the optimal policy. Our
formulation results in a efficient algorithm that accounts for a simple
re-weighting of policy actions in the standard policy iteration scheme. We
extend our approach to continuous state/action space and present a practical
algorithm, distributionally robust soft actor-critic, that implements a
different exploration strategy: it acts conservatively at short-term and it
explores optimistically in a long-run. We provide promising experimental
results on continuous control tasks.
