  Molecular property prediction (MPP) is a crucial task in the drug discovery
pipeline, which has recently gained considerable attention thanks to advances
in deep neural networks. However, recent research has revealed that deep models
struggle to beat traditional non-deep ones on MPP. In this study, we benchmark
12 representative models (3 non-deep models and 9 deep models) on 14 molecule
datasets. Through the most comprehensive study to date, we make the following
key observations: \textbf{(\romannumeral 1)} Deep models are generally unable
to outperform non-deep ones; \textbf{(\romannumeral 2)} The failure of deep
models on MPP cannot be solely attributed to the small size of molecular
datasets. What matters is the irregular molecule data pattern;
\textbf{(\romannumeral 3)} In particular, tree models using molecular
fingerprints as inputs tend to perform better than other competitors.
Furthermore, we conduct extensive empirical investigations into the unique
patterns of molecule data and inductive biases of various models underlying
these phenomena.
