  Multi-Agent Path Finding (MAPF) aims to arrange collision-free goal-reaching
paths for a group of agents. Anytime MAPF solvers based on large neighborhood
search (LNS) have gained prominence recently due to their flexibility and
scalability. Neighborhood selection strategy is crucial to the success of
MAPF-LNS and a flurry of methods have been proposed. However, several pitfalls
exist and hinder a comprehensive evaluation of these new methods, which mainly
include: 1) Lower than actual or incorrect baseline performance; 2) Lack of a
unified evaluation setting and criterion; 3) Lack of a codebase or executable
model for supervised learning methods. To overcome these challenges, we conduct
a fair comparison across prominent methods on the same benchmark and
hyperparameter search settings. Additionally, we propose a simple neighborhood
selection strategy which marks a clear advancement in terms of runtime
efficiency in large maps with large number of agents. Our benchmarking
evaluation promotes new challenges for existing learning based methods and
presents opportunities for future research when machine learning is integrated
with MAPF-LNS. Code and data are available at
https://github.com/ChristinaTan0704/mapf-lns-benchmark.
