  More research attention has recently been given to end-to-end autonomous
driving technologies where the entire driving pipeline is replaced with a
single neural network because of its simpler structure and faster inference
time. Despite this appealing approach largely reducing the components in the
driving pipeline, its simplicity also leads to interpretability problems and
safety issues. The trained policy is not always compliant with the traffic
rules and it is also hard to discover the reason for the misbehavior because of
the lack of intermediate outputs. Meanwhile, sensors are also critical to
autonomous driving's security and feasibility to perceive the surrounding
environment under complex driving scenarios. In this paper, we proposed P-CSG,
a penalty-based imitation learning approach with cross semantics generation
sensor fusion technologies to increase the overall performance of end-to-end
autonomous driving. In this method, we introduce three penalties - red light,
stop sign, and curvature speed penalty to make the agent more sensitive to
traffic rules. The proposed cross semantics generation helps to align the
shared information from different input modalities. We assessed our model's
performance using the CARLA leaderboard - Town 05 Long benchmark and Longest6
Benchmark, achieving an impressive driving score improvement. Furthermore, we
conducted robustness evaluations against adversarial attacks like FGSM and Dot
attacks, revealing a substantial increase in robustness compared to baseline
models. More detailed information, such as code base resources, and videos can
be found at https://hk-zh.github.io/p-csg-plus.
