  In end-to-end autonomous driving, the utilization of existing sensor fusion
techniques and navigational control methods for imitation learning proves
inadequate in challenging situations that involve numerous dynamic agents. To
address this issue, we introduce LeTFuser, a lightweight transformer-based
algorithm for fusing multiple RGB-D camera representations. To perform
perception and control tasks simultaneously, we utilize multi-task learning.
Our model comprises of two modules, the first being the perception module that
is responsible for encoding the observation data obtained from the RGB-D
cameras. Our approach employs the Convolutional vision Transformer (CvT)
\cite{wu2021cvt} to better extract and fuse features from multiple RGB cameras
due to local and global feature extraction capability of convolution and
transformer modules, respectively. Encoded features combined with static and
dynamic environments are later employed by our control module to predict
waypoints and vehicular controls (e.g. steering, throttle, and brake). We use
two methods to generate the vehicular controls levels. The first method uses a
PID algorithm to follow the waypoints on the fly, whereas the second one
directly predicts the control policy using the measurement features and
environmental state. We evaluate the model and conduct a comparative analysis
with recent models on the CARLA simulator using various scenarios, ranging from
normal to adversarial conditions, to simulate real-world scenarios. Our method
demonstrated better or comparable results with respect to our baselines in term
of driving abilities. The code is available at
\url{https://github.com/pagand/e2etransfuser/tree/cvpr-w} to facilitate future
studies.
