  In the domain of autonomous driving, the offline Reinforcement Learning~(RL)
approaches exhibit notable efficacy in addressing sequential decision-making
problems from offline datasets. However, maintaining safety in diverse
safety-critical scenarios remains a significant challenge due to long-tailed
and unforeseen scenarios absent from offline datasets. In this paper, we
introduce the saFety-aware strUctured Scenario representatION (FUSION), a
pioneering representation learning method in offline RL to facilitate the
learning of a generalizable end-to-end driving policy by leveraging structured
scenario information. FUSION capitalizes on the causal relationships between
the decomposed reward, cost, state, and action space, constructing a framework
for structured sequential reasoning in dynamic traffic environments. We conduct
extensive evaluations in two typical real-world settings of the distribution
shift in autonomous vehicles, demonstrating the good balance between safety
cost and utility reward compared to the current state-of-the-art safe RL and IL
baselines. Empirical evidence in various driving scenarios attests that FUSION
significantly enhances the safety and generalizability of autonomous driving
agents, even in the face of challenging and unseen environments. Furthermore,
our ablation studies reveal noticeable improvements in the integration of
causal representation into the offline safe RL algorithm. Our code
implementation is available at: https://sites.google.com/view/safe-fusion/.
