  A central challenge in the development of nanophotonic structures is
identifying the optimal design for a target functionality, and understanding
the physical mechanisms that enable the optimized device's capabilities.
Previously investigated design methods for nanophotonic structures, including
both conventional optimization approaches as well as nascent machine learning
(ML) strategies, have made progress, yet they remain 'black boxes' that lack
explanations for their predictions. Here we demonstrate that convolutional
neural networks (CNN) trained to predict the electromagnetic response of
classes of metal-dielectric-metal metamaterials, including complex freeform
designs, can be explained to reveal deeper insights into the underlying physics
of nanophotonic structures. Using an explainable AI (XAI) approach, we show
that we can identify the importance of specific spatial regions of a
nanophotonic structure for the presence or lack of an absorption peak. Our
results highlight that ML strategies can be used for physics discovery, as well
as design optimization, in optics and photonics.
