  Introduction: The presence of fibrillatory waves (f-waves) is important in
the diagnosis of atrial fibrillation (AF), which has motivated the development
of methods for f-wave extraction. We propose a novel approach to benchmarking
methods designed for single-lead ECG analysis, building on the hypothesis that
better-performing AF classification using features computed from the extracted
f-waves implies better-performing extraction. The approach is well-suited for
processing large Holter data sets annotated with respect to the presence of AF.
Methods: Three data sets with a total of 300 two- or three-lead Holter
recordings, performed in the USA, Israel and Japan, were used as well as a
simulated single-lead data set. Four existing extraction methods based on
either average beat subtraction or principal component analysis (PCA) were
evaluated. A random forest classifier was used for window-based AF
classification. Performance was measured by the area under the receiver
operating characteristic (AUROC). Results: The best performance was found for
PCA-based extraction, resulting in AUROCs in the ranges 0.77--0.83, 0.62--0.78,
and 0.87--0.89 for the data sets from USA, Israel, and Japan, respectively,
when analyzed across leads; the AUROC of the simulated single-lead, noisy data
set was 0.98. Conclusions: This study provides a novel approach to evaluating
the performance of f-wave extraction methods, offering the advantage of not
using ground truth f-waves for evaluation, thus being able to leverage real
data sets for evaluation. The code is open source (following publication).
