  On-policy reinforcement learning methods, like Trust Region Policy
Optimization (TRPO) and Proximal Policy Optimization (PPO), often demand
extensive data per update, leading to sample inefficiency. This paper
introduces Reflective Policy Optimization (RPO), a novel on-policy extension
that amalgamates past and future state-action information for policy
optimization. This approach empowers the agent for introspection, allowing
modifications to its actions within the current state. Theoretical analysis
confirms that policy performance is monotonically improved and contracts the
solution space, consequently expediting the convergence procedure. Empirical
results demonstrate RPO's feasibility and efficacy in two reinforcement
learning benchmarks, culminating in superior sample efficiency. The source code
of this work is available at https://github.com/Edgargan/RPO.
