  Imagining potential outcomes of actions before execution helps agents make
more informed decisions, a prospective thinking ability fundamental to human
cognition. However, mainstream model-free Reinforcement Learning (RL) methods
lack the ability to proactively envision future scenarios, plan, and guide
strategies. These methods typically rely on trial and error to adjust policy
functions, aiming to maximize cumulative rewards or long-term value, even if
such high-reward decisions place the environment in extremely dangerous states.
To address this, we propose the Prospective (ProSpec) RL method, which makes
higher-value, lower-risk optimal decisions by imagining future n-stream
trajectories. Specifically, ProSpec employs a dynamic model to predict future
states (termed "imagined states") based on the current state and a series of
sampled actions. Furthermore, we integrate the concept of Model Predictive
Control and introduce a cycle consistency constraint that allows the agent to
evaluate and select the optimal actions from these trajectories. Moreover,
ProSpec employs cycle consistency to mitigate two fundamental issues in RL:
augmenting state reversibility to avoid irreversible events (low risk) and
augmenting actions to generate numerous virtual trajectories, thereby improving
data efficiency. We validated the effectiveness of our method on the DMControl
benchmarks, where our approach achieved significant performance improvements.
Code will be open-sourced upon acceptance.
