  Risk-sensitive reinforcement learning (RL) aims to optimize policies that
balance the expected reward and risk. In this paper, we present a novel
risk-sensitive RL framework that employs an Iterated Conditional Value-at-Risk
(CVaR) objective under both linear and general function approximations,
enriched by human feedback. These new formulations provide a principled way to
guarantee safety in each decision making step throughout the control process.
Moreover, integrating human feedback into risk-sensitive RL framework bridges
the gap between algorithmic decision-making and human participation, allowing
us to also guarantee safety for human-in-the-loop systems. We propose provably
sample-efficient algorithms for this Iterated CVaR RL and provide rigorous
theoretical analysis. Furthermore, we establish a matching lower bound to
corroborate the optimality of our algorithms in a linear context.
