  Current advancements in reinforcement learning (RL) have predominantly
focused on learning step-based policies that generate actions for each
perceived state. While these methods efficiently leverage step information from
environmental interaction, they often ignore the temporal correlation between
actions, resulting in inefficient exploration and unsmooth trajectories that
are challenging to implement on real hardware. Episodic RL (ERL) seeks to
overcome these challenges by exploring in parameters space that capture the
correlation of actions. However, these approaches typically compromise data
efficiency, as they treat trajectories as opaque \emph{black boxes}. In this
work, we introduce a novel ERL algorithm, Temporally-Correlated Episodic RL
(TCE), which effectively utilizes step information in episodic policy updates,
opening the 'black box' in existing ERL methods while retaining the smooth and
consistent exploration in parameter space. TCE synergistically combines the
advantages of step-based and episodic RL, achieving comparable performance to
recent ERL methods while maintaining data efficiency akin to state-of-the-art
(SoTA) step-based RL.
