  Addressing hard cases in autonomous driving, such as anomalous road users,
extreme weather conditions, and complex traffic interactions, presents
significant challenges. To ensure safety, it is crucial to detect and manage
these scenarios effectively for autonomous driving systems. However, the rarity
and high-risk nature of these cases demand extensive, diverse datasets for
training robust models. Vision-Language Foundation Models (VLMs) have shown
remarkable zero-shot capabilities as being trained on extensive datasets. This
work explores the potential of VLMs in detecting hard cases in autonomous
driving. We demonstrate the capability of VLMs such as GPT-4v in detecting hard
cases in traffic participant motion prediction on both agent and scenario
levels. We introduce a feasible pipeline where VLMs, fed with sequential image
frames with designed prompts, effectively identify challenging agents or
scenarios, which are verified by existing prediction models. Moreover, by
taking advantage of this detection of hard cases by VLMs, we further improve
the training efficiency of the existing motion prediction pipeline by
performing data selection for the training samples suggested by GPT. We show
the effectiveness and feasibility of our pipeline incorporating VLMs with
state-of-the-art methods on NuScenes datasets. The code is accessible at
https://github.com/KTH-RPL/Detect_VLM.
