  Machine learning has emerged as a new tool in chemistry to bypass expensive
experiments or quantum-chemical calculations, for example, in high-throughput
screening applications. However, many machine learning studies rely on small
data sets, making it difficult to efficiently implement powerful deep learning
architectures such as message passing neural networks. In this study, we
benchmark common machine learning models for the prediction of molecular
properties on small data sets, for which the best results are obtained with the
message passing neural network PaiNN, as well as SOAP molecular descriptors
concatenated to a set of simple molecular descriptors tailored to gradient
boosting with regression trees. To further improve the predictive capabilities
of PaiNN, we present a transfer learning strategy that uses large data sets to
pre-train the respective models and allows to obtain more accurate models after
fine-tuning on the original data sets. The pre-training labels are obtained
from computationally cheap ab initio or semi-empirical models and corrected by
simple linear regression on the target data set to obtain labels that are close
to those of the original data. This strategy is tested on the Harvard Oxford
Photovoltaics data set (HOPV, HOMO-LUMO-gaps), for which excellent results are
obtained, and on the Freesolv data set (solvation energies), where this method
is unsuccessful due to a complex underlying learning task and the dissimilar
methods used to obtain pre-training and fine-tuning labels. Finally, we find
that the final training results do not improve monotonically with the size of
the pre-training data set, but pre-training with fewer data points can lead to
more biased pre-trained models and higher accuracy after fine-tuning.
