  Combining offline and online reinforcement learning (RL) is crucial for
efficient and safe learning. However, previous approaches treat offline and
online learning as separate procedures, resulting in redundant designs and
limited performance. We ask: Can we achieve straightforward yet effective
offline and online learning without introducing extra conservatism or
regularization? In this study, we propose Uni-o4, which utilizes an on-policy
objective for both offline and online learning. Owning to the alignment of
objectives in two phases, the RL agent can transfer between offline and online
learning seamlessly. This property enhances the flexibility of the learning
paradigm, allowing for arbitrary combinations of pretraining, fine-tuning,
offline, and online learning. In the offline phase, specifically, Uni-o4
leverages diverse ensemble policies to address the mismatch issues between the
estimated behavior policy and the offline dataset. Through a simple offline
policy evaluation (OPE) approach, Uni-o4 can achieve multi-step policy
improvement safely. We demonstrate that by employing the method above, the
fusion of these two paradigms can yield superior offline initialization as well
as stable and rapid online fine-tuning capabilities. Through real-world robot
tasks, we highlight the benefits of this paradigm for rapid deployment in
challenging, previously unseen real-world environments. Additionally, through
comprehensive evaluations using numerous simulated benchmarks, we substantiate
that our method achieves state-of-the-art performance in both offline and
offline-to-online fine-tuning learning. Our website:
https://lei-kun.github.io/uni-o4/ .
