  Safe Reinforcement Learning (RL) aims to find a policy that achieves high
rewards while satisfying cost constraints. When learning from scratch, safe RL
agents tend to be overly conservative, which impedes exploration and restrains
the overall performance. In many realistic tasks, e.g. autonomous driving,
large-scale expert demonstration data are available. We argue that extracting
expert policy from offline data to guide online exploration is a promising
solution to mitigate the conserveness issue. Large-capacity models, e.g.
decision transformers (DT), have been proven to be competent in offline policy
learning. However, data collected in real-world scenarios rarely contain
dangerous cases (e.g., collisions), which makes it prohibitive for the policies
to learn safety concepts. Besides, these bulk policy networks cannot meet the
computation speed requirements at inference time on real-world tasks such as
autonomous driving. To this end, we propose Guided Online Distillation (GOLD),
an offline-to-online safe RL framework. GOLD distills an offline DT policy into
a lightweight policy network through guided online safe RL training, which
outperforms both the offline DT policy and online safe RL algorithms.
Experiments in both benchmark safe RL tasks and real-world driving tasks based
on the Waymo Open Motion Dataset (WOMD) demonstrate that GOLD can successfully
distill lightweight policies and solve decision-making problems in challenging
safety-critical scenarios.
