  Virtual Screening (VS) of vast compound libraries guided by Artificial
Intelligence (AI) models is a highly productive approach to early drug
discovery. Data splitting is crucial for better benchmarking of such AI models.
Traditional random data splits produce similar molecules between training and
test sets, conflicting with the reality of VS libraries which mostly contain
structurally distinct compounds. Scaffold split, grouping molecules by shared
core structure, is widely considered to reflect this real-world scenario.
However, here we show that the scaffold split also overestimates VS
performance. The reason is that molecules with different chemical scaffolds are
often similar, which hence introduces unrealistically high similarities between
training molecules and test molecules following a scaffold split. Our study
examined three representative AI models on 60 NCI-60 datasets, each with
approximately 30,000 to 50,000 molecules tested on a different cancer cell
line. Each dataset was split with three methods: scaffold, Butina clustering
and the more accurate Uniform Manifold Approximation and Projection (UMAP)
clustering. Regardless of the model, model performance is much worse with UMAP
splits from the results of the 2100 models trained and evaluated for each
algorithm and split. These robust results demonstrate the need for more
realistic data splits to tune, compare, and select models for VS. For the same
reason, avoiding the scaffold split is also recommended for other molecular
property prediction problems. The code to reproduce these results is available
at https://github.com/ScaffoldSplitsOverestimateVS
