  Handling long tail corner cases is a major challenge faced by autonomous
vehicles (AVs). While large language models (LLMs) hold great potentials to
handle the corner cases with excellent generalization and explanation
capabilities and received increasing research interest on application to
autonomous driving, there are still technical barriers to be tackled, such as
strict model performance and huge computing resource requirements of LLMs. In
this paper, we investigate a new approach of applying remote or edge LLMs to
support autonomous driving. A key issue for such LLM assisted driving system is
the assessment of LLMs on their understanding of driving theory and skills,
ensuring they are qualified to undertake safety critical driving assistance
tasks for CAVs. We design and run driving theory tests for several proprietary
LLM models (OpenAI GPT models, Baidu Ernie and Ali QWen) and open-source LLM
models (Tsinghua MiniCPM-2B and MiniCPM-Llama3-V2.5) with more than 500
multiple-choices theory test questions. Model accuracy, cost and processing
latency are measured from the experiments. Experiment results show that while
model GPT-4 passes the test with improved domain knowledge and Ernie has an
accuracy of 85% (just below the 86% passing threshold), other LLM models
including GPT-3.5 fail the test. For the test questions with images, the
multimodal model GPT4-o has an excellent accuracy result of 96%, and the
MiniCPM-Llama3-V2.5 achieves an accuracy of 76%. While GPT-4 holds stronger
potential for CAV driving assistance applications, the cost of using model GPT4
is much higher, almost 50 times of that of using GPT3.5. The results can help
make decision on the use of the existing LLMs for CAV applications and
balancing on the model performance and cost.
