  Using publicly available data to determine the performance of methodological
contributions is important as it facilitates reproducibility and allows
scrutiny of the published results. In lung nodule classification, for example,
many works report results on the publicly available LIDC dataset. In theory,
this should allow a direct comparison of the performance of proposed methods
and assess the impact of individual contributions. When analyzing seven recent
works, however, we find that each employs a different data selection process,
leading to largely varying total number of samples and ratios between benign
and malignant cases. As each subset will have different characteristics with
varying difficulty for classification, a direct comparison between the proposed
methods is thus not always possible, nor fair. We study the particular effect
of truthing when aggregating labels from multiple experts. We show that
specific choices can have severe impact on the data distribution where it may
be possible to achieve superior performance on one sample distribution but not
on another. While we show that we can further improve on the state-of-the-art
on one sample selection, we also find that on a more challenging sample
selection, on the same database, the more advanced models underperform with
respect to very simple baseline methods, highlighting that the selected data
distribution may play an even more important role than the model architecture.
This raises concerns about the validity of claimed methodological
contributions. We believe the community should be aware of these pitfalls and
make recommendations on how these can be avoided in future work.
