  Model-based Reinforcement Learning (RL) has shown its high training
efficiency and capability of handling high-dimensional tasks. Regarding safety
issues, safe model-based RL can achieve nearly zero-cost performance and
effectively manage the trade-off between performance and safety. Nevertheless,
prior works still pose safety challenges due to the online exploration in
real-world deployment. To address this, some offline RL methods have emerged as
solutions, which learn from a static dataset in a safe way by avoiding
interactions with the environment. In this paper, we aim to further enhance
safety during the deployment stage for vision-based robotic tasks by
fine-tuning an offline-trained policy. We incorporate in-sample optimization,
model-based policy expansion, and reachability guidance to construct a safe
offline-to-online framework. Moreover, our method proves to improve the
generalization of offline policy in unseen safety-constrained scenarios.
Finally, the efficiency of our method is validated on simulation benchmarks
with five vision-only tasks and a real robot by solving some deployment
problems using limited data.
