  The use of reinforcement learning (RL) in practical applications requires
considering sub-optimal outcomes, which depend on the agent's familiarity with
the uncertain environment. Dynamically adjusting the level of epistemic risk
over the course of learning can tactically achieve reliable optimal policy in
safety-critical environments and tackle the sub-optimality of a static risk
level. In this work, we introduce a novel framework, Distributional RL with
Online Risk Adaption (DRL-ORA), which can quantify the aleatory and epistemic
uncertainties compositely and dynamically select the epistemic risk levels via
solving a total variation minimization problem online. The risk level selection
can be efficiently achieved through grid search using a Follow-The-Leader type
algorithm, and its offline oracle is related to "satisficing measure" (in the
decision analysis community) under a special modification of the loss function.
We show multiple classes of tasks where DRL-ORA outperforms existing methods
that rely on either a fixed risk level or manually predetermined risk level
adaption. Given the simplicity of our modifications, we believe the framework
can be easily incorporated into most RL algorithm variants.
