  The goal of most materials discovery is to discover materials that are
superior to those currently known. Fundamentally, this is close to
extrapolation, which is a weak point for most machine learning models that
learn the probability distribution of data. Herein, we develop reinforcement
learning-guided combinatorial chemistry, which is a rule-based molecular
designer driven by trained policy for selecting subsequent molecular fragments
to get a target molecule. Since our model has the potential to generate all
possible molecular structures that can be obtained from combinations of
molecular fragments, unknown molecules with superior properties can be
discovered. We theoretically and empirically demonstrate that our model is more
suitable for discovering better compounds than probability
distribution-learning models. In an experiment aimed at discovering molecules
that hit seven extreme target properties, our model discovered 1,315 of all
target-hitting molecules and 7,629 of five target-hitting molecules out of
100,000 trials, whereas the probability distribution-learning models failed.
Moreover, it has been confirmed that every molecule generated under the binding
rules of molecular fragments is 100% chemically valid. To illustrate the
performance in actual problems, we also demonstrate that our models work well
on two practical applications: discovering protein docking molecules and HIV
inhibitors.
