  Recent breakthroughs by deep neural networks (DNNs) in natural language
processing (NLP) and computer vision have been driven by a scale-up of models
and data rather than the discovery of novel computing paradigms. Here, we
investigate if scale can have a similar impact for models designed to aid small
molecule drug discovery. We address this question through a large-scale and
systematic analysis of how DNN size, data diet, and learning routines interact
to impact accuracy on our Phenotypic Chemistry Arena (Pheno-CA) benchmark: a
diverse set of drug development tasks posed on image-based high content
screening data. Surprisingly, we find that DNNs explicitly supervised to solve
tasks in the Pheno-CA do not continuously improve as their data and model size
is scaled-up. To address this issue, we introduce a novel precursor task, the
Inverse Biological Process (IBP), which is designed to resemble the causal
objective functions that have proven successful for NLP. We indeed find that
DNNs first trained with IBP then probed for performance on the Pheno-CA
significantly outperform task-supervised DNNs. More importantly, the
performance of these IBP-trained DNNs monotonically improves with data and
model scale. Our findings reveal that the DNN ingredients needed to accurately
solve small molecule drug development tasks are already in our hands, and
project how much more experimental data is needed to achieve any desired level
of improvement. We release our Pheno-CA benchmark and code to encourage further
study of neural scaling laws for small molecule drug discovery.
