  Rapid advancements in Autonomous Driving (AD) tasks turned a significant
shift toward end-to-end fashion, particularly in the utilization of
vision-language models (VLMs) that integrate robust logical reasoning and
cognitive abilities to enable comprehensive end-to-end planning. However, these
VLM-based approaches tend to integrate 2D vision tokenizers and a large
language model (LLM) for ego-car planning, which lack 3D geometric priors as a
cornerstone of reliable planning. Naturally, this observation raises a critical
concern: Can a 2D-tokenized LLM accurately perceive the 3D environment? Our
evaluation of current VLM-based methods across 3D object detection, vectorized
map construction, and environmental caption suggests that the answer is,
unfortunately, NO. In other words, 2D-tokenized LLM fails to provide reliable
autonomous driving. In response, we introduce DETR-style 3D perceptrons as 3D
tokenizers, which connect LLM with a one-layer linear projector. This simple
yet elegant strategy, termed Atlas, harnesses the inherent priors of the 3D
physical world, enabling it to simultaneously process high-resolution
multi-view images and employ spatiotemporal modeling. Despite its simplicity,
Atlas demonstrates superior performance in both 3D detection and ego planning
tasks on nuScenes dataset, proving that 3D-tokenized LLM is the key to reliable
autonomous driving. The code and datasets will be released.
