  Inverse design coupled with adjoint optimization is a powerful method to
design on-chip nanophotonic devices with multi-wavelength and multi-mode
optical functionalities. Although only two simulations are required in each
iteration of this optimization process, these simulations still make up the
vast majority of the necessary computations, and render the design of complex
devices with large footprints computationally infeasible. Here, we introduce a
multi-faceted factorization caching approach to drastically simplify the
underlying computations in finite-difference frequency-domain (FDFD)
simulations, and significantly reduce the time required for device
optimization. Specifically, we cache the symbolic and numerical factorizations
for the solution of the corresponding system of linear equations in discretized
FDFD simulations, and re-use them throughout the entire device design process.
As proof-of-concept demonstrations of the resulting computational advantage, we
present simulation speedups reaching as high as $9.2\times$ in the design of
broadband wavelength and mode multiplexers compared to conventional FDFD
methods. We also show that factorization caching scales well over a broad range
of footprints independent of the device geometry, from as small as $16{\mu
m}^2$ to over $7000 {\mu m}^2$. Our results present significant enhancements in
the computational efficiency of inverse photonic design, and can greatly
accelerate the use of machine-optimized devices in future photonic systems.
