  In the field of optics, precise control of light with arbitrary spatial
resolution has long been a sought-after goal. Freeform nanophotonic devices are
critical building blocks for achieving this goal, as they provide access to a
design potential that could hardly be achieved by conventional fixed-shape
devices. However, finding an optimal device structure in the vast combinatorial
design space that scales exponentially with the number of freeform design
parameters has been an enormous challenge. In this study, we propose
physics-informed reinforcement learning (PIRL) as an optimization method for
freeform nanophotonic devices, which combines the adjoint-based method with
reinforcement learning to enhance the sample efficiency of the optimization
algorithm and overcome the issue of local minima. To illustrate these
advantages of PIRL over other conventional optimization algorithms, we design a
family of one-dimensional metasurface beam deflectors using PIRL, obtaining
more performant devices. We also explore the transfer learning capability of
PIRL that further improves sample efficiency and demonstrate how the minimum
feature size of the design can be enforced in PIRL through reward engineering.
With its high sample efficiency, robustness, and ability to seamlessly
incorporate practical device design constraints, our method offers a promising
approach to highly combinatorial freeform device optimization in various
physical domains.
