  In recent times, there has been a growing focus on end-to-end autonomous
driving technologies. This technology involves the replacement of the entire
driving pipeline with a single neural network, which has a simpler structure
and faster inference time. However, while this approach reduces the number of
components in the driving pipeline, it also presents challenges related to
interpretability and safety. For instance, the trained policy may not always
comply with traffic rules, and it is difficult to determine the reason for such
misbehavior due to the lack of intermediate outputs. Additionally, the
successful implementation of autonomous driving technology heavily depends on
the reliable and expedient processing of sensory data to accurately perceive
the surrounding environment. In this paper, we provide penalty-based imitation
learning approach combined with cross semantics generation sensor fusion
technologies (P-CSG) to efficiently integrate multiple modalities of
information and enable the autonomous agent to effectively adhere to traffic
regulations. Our model undergoes evaluation within the Town 05 Long benchmark,
where we observe a remarkable increase in the driving score by more than 12%
when compared to the state-of-the-art (SOTA) model, InterFuser. Notably, our
model achieves this performance enhancement while achieving a 7-fold increase
in inference speed and reducing the model size by approximately 30%. For more
detailed information, including code-based resources, they can be found at
https://hk-zh.github.io/p-csg/
