  The autonomous driving industry is increasingly adopting end-to-end learning
from sensory inputs to minimize human biases in system design. Traditional
end-to-end driving models, however, suffer from long-tail events due to rare or
unseen inputs within their training distributions. To address this, we propose
TOKEN, a novel Multi-Modal Large Language Model (MM-LLM) that tokenizes the
world into object-level knowledge, enabling better utilization of LLM's
reasoning capabilities to enhance autonomous vehicle planning in long-tail
scenarios. TOKEN effectively alleviates data scarcity and inefficient
tokenization by leveraging a traditional end-to-end driving model to produce
condensed and semantically enriched representations of the scene, which are
optimized for LLM planning compatibility through deliberate representation and
reasoning alignment training stages. Our results demonstrate that TOKEN excels
in grounding, reasoning, and planning capabilities, outperforming existing
frameworks with a 27% reduction in trajectory L2 error and a 39% decrease in
collision rates in long-tail scenarios. Additionally, our work highlights the
importance of representation alignment and structured reasoning in sparking the
common-sense reasoning capabilities of MM-LLMs for effective planning.
