  Aiding humans with scientific designs is one of the most exciting of
artificial intelligence (AI) and machine learning (ML), due to their potential
for the discovery of new drugs, design of new materials and chemical compounds,
etc. However, scientific design typically requires complex domain knowledge
that is not familiar to AI researchers. Further, scientific studies involve
professional skills to perform experiments and evaluations. These obstacles
prevent AI researchers from developing specialized methods for scientific
designs. To take a step towards easy-to-understand and reproducible research of
scientific design, we propose a benchmark for the inverse design of
nanophotonic devices, which can be verified computationally and accurately.
Specifically, we implemented three different nanophotonic design problems,
namely a radiative cooler, a selective emitter for thermophotovoltaics, and
structural color filters, all of which are different in design parameter
spaces, complexity, and design targets. The benchmark environments are
implemented with an open-source simulator. We further implemented 10 different
inverse design algorithms and compared them in a reproducible and fair
framework. The results revealed the strengths and weaknesses of existing
methods, which shed light on several future directions for developing more
efficient inverse design algorithms. Our benchmark can also serve as the
starting point for more challenging scientific design problems. The code of
IDToolkit is available at https://github.com/ThyrixYang/IDToolkit.
