  Motivation: Accurate estimation of false discovery rate (FDR) of spectral
identification is a central problem in mass spectrometry-based proteomics. Over
the past two decades, target decoy approaches (TDAs) and decoy-free approaches
(DFAs), have been widely used to estimate FDR. TDAs use a database of decoy
species to faithfully model score distributions of incorrect peptide-spectrum
matches (PSMs). DFAs, on the other hand, fit two-component mixture models to
learn the parameters of correct and incorrect PSM score distributions. While
conceptually straightforward, both approaches lead to problems in practice,
particularly in experiments that push instrumentation to the limit and generate
low fragmentation-efficiency and low signal-to-noise-ratio spectra. Results: We
introduce a new decoy-free framework for FDR estimation that generalizes
present DFAs while exploiting more search data in a manner similar to TDAs. Our
approach relies on multi-component mixtures, in which score distributions
corresponding to the correct PSMs, best incorrect PSMs, and second-best
incorrect PSMs are modeled by the skew normal family. We derive EM algorithms
to estimate parameters of these distributions from the scores of best and
second-best PSMs associated with each experimental spectrum. We evaluate our
models on multiple proteomics datasets and a HeLa cell digest case study
consisting of more than a million spectra in total. We provide evidence of
improved performance over existing DFAs and improved stability and speed over
TDAs without any performance degradation. We propose that the new strategy has
the potential to extend beyond peptide identification and reduce the need for
TDA on all analytical platforms.
