  Lithium niobate-on-insulator (LNOI) is an emerging photonic platform that
exhibits favorable material properties (such as low optical loss, strong
nonlinearities, and stability) and enables large-scale integration with
stronger optical confinement, showing promise for future optical networks,
quantum processors, and nonlinear optical systems. However, while photonics
engineering has entered the era of automated "inverse design" via optimization
in recent years, the design of LNOI integrated photonic devices still mostly
relies on intuitive models and inefficient parameter sweeps, limiting the
accessible parameter space, performance, and functionality. Here, we develop
and implement a 3D gradient-based inverse-design model tailored for topology
optimization of the LNOI platform, which not only could efficiently search a
large parameter space but also takes into account practical fabrication
constraints, including minimum feature sizes and etched sidewall angles. We
experimentally demonstrate a spatial-mode multiplexer, a waveguide crossing,
and a compact waveguide bend, all with low insertion losses, tiny footprints,
and excellent agreement between simulation and experimental results. The
devices, together with the design methodology, represent a crucial step towards
the variety of advanced device functionalities needed in future LNOI photonics,
and could provide compact and cost-effective solutions for future optical
links, quantum technologies and nonlinear optics.
