  Crystal structure prediction (CSP) is now increasingly used in discovering
novel materials with applications in diverse industries. However, despite
decades of developments and significant progress in this area, there lacks a
set of well-defined benchmark dataset, quantitative performance metrics, and
studies that evaluate the status of the field. We aim to fill this gap by
introducing a CSP benchmark suite with 180 test structures along with our
recently implemented CSP performance metric set. We benchmark a collection of
13 state-of-the-art (SOTA) CSP algorithms including template-based CSP
algorithms, conventional CSP algorithms based on DFT calculations and global
search such as CALYPSO, CSP algorithms based on machine learning (ML)
potentials and global search, and distance matrix based CSP algorithms. Our
results demonstrate that the performance of the current CSP algorithms is far
from being satisfactory. Most algorithms cannot even identify the structures
with the correct space groups except for the template-based algorithms when
applied to test structures with similar templates. We also find that the ML
potential based CSP algorithms are now able to achieve competitive performances
compared to the DFT-based algorithms. These CSP algorithms' performance is
strongly determined by the quality of the neural potentials as well as the
global optimization algorithms. Our benchmark suite comes with a comprehensive
open-source codebase and 180 well-selected benchmark crystal structures, making
it convenient to evaluate the advantages and disadvantages of CSP algorithms
from future studies. All the code and benchmark data are available at
https://github.com/usccolumbia/cspbenchmark
