  Predicting the future motion of surrounding agents is essential for
autonomous vehicles (AVs) to operate safely in dynamic, human-robot-mixed
environments. Context information, such as road maps and surrounding agents'
states, provides crucial geometric and semantic information for motion behavior
prediction. To this end, recent works explore two-stage prediction frameworks
where coarse trajectories are first proposed, and then used to select critical
context information for trajectory refinement. However, they either incur a
large amount of computation or bring limited improvement, if not both. In this
paper, we introduce a novel scenario-adaptive refinement strategy, named
SmartRefine, to refine prediction with minimal additional computation.
Specifically, SmartRefine can comprehensively adapt refinement configurations
based on each scenario's properties, and smartly chooses the number of
refinement iterations by introducing a quality score to measure the prediction
quality and remaining refinement potential of each scenario. SmartRefine is
designed as a generic and flexible approach that can be seamlessly integrated
into most state-of-the-art motion prediction models. Experiments on Argoverse
(1 & 2) show that our method consistently improves the prediction accuracy of
multiple state-of-the-art prediction models. Specifically, by adding
SmartRefine to QCNet, we outperform all published ensemble-free works on the
Argoverse 2 leaderboard (single agent track) at submission. Comprehensive
studies are also conducted to ablate design choices and explore the mechanism
behind multi-iteration refinement. Codes are available at
https://github.com/opendilab/SmartRefine/
