  Tandem mass spectrometry has played a pivotal role in advancing proteomics,
enabling the high-throughput analysis of protein composition in biological
tissues. Many deep learning methods have been developed for \emph{de novo}
peptide sequencing task, i.e., predicting the peptide sequence for the observed
mass spectrum. However, two key challenges seriously hinder the further
advancement of this important task. Firstly, since there is no consensus for
the evaluation datasets, the empirical results in different research papers are
often not comparable, leading to unfair comparison. Secondly, the current
methods are usually limited to amino acid-level or peptide-level precision and
recall metrics. In this work, we present the first unified benchmark NovoBench
for \emph{de novo} peptide sequencing, which comprises diverse mass spectrum
data, integrated models, and comprehensive evaluation metrics. Recent
impressive methods, including DeepNovo, PointNovo, Casanovo, InstaNovo, AdaNovo
and $\pi$-HelixNovo are integrated into our framework. In addition to amino
acid-level and peptide-level precision and recall, we evaluate the models'
performance in terms of identifying post-tranlational modifications (PTMs),
efficiency and robustness to peptide length, noise peaks and missing fragment
ratio, which are important influencing factors while seldom be considered.
Leveraging this benchmark, we conduct a large-scale study of current methods,
report many insightful findings that open up new possibilities for future
development. The benchmark will be open-sourced to facilitate future research
and application.
