  Automatic static analysis tools (ASATs), such as Findbugs, have a high false
alarm rate. The large number of false alarms produced poses a barrier to
adoption. Researchers have proposed the use of machine learning to prune false
alarms and present only actionable warnings to developers. The state-of-the-art
study has identified a set of "Golden Features" based on metrics computed over
the characteristics and history of the file, code, and warning. Recent studies
show that machine learning using these features is extremely effective and that
they achieve almost perfect performance.
  We perform a detailed analysis to better understand the strong performance of
the "Golden Features". We found that several studies used an experimental
procedure that results in data leakage and data duplication, which are subtle
issues with significant implications. Firstly, the ground-truth labels have
leaked into features that measure the proportion of actionable warnings in a
given context. Secondly, many warnings in the testing dataset appear in the
training dataset. Next, we demonstrate limitations in the warning oracle that
determines the ground-truth labels, a heuristic comparing warnings in a given
revision to a reference revision in the future. We show the choice of reference
revision influences the warning distribution. Moreover, the heuristic produces
labels that do not agree with human oracles. Hence, the strong performance of
these techniques previously seen is overoptimistic of their true performance if
adopted in practice. Our results convey several lessons and provide guidelines
for evaluating false alarm detectors.
