  Photonic inverse design has emerged as an indispensable engineering tool for
complex optical systems. In many instances it is important to optimize for both
material and geometry configurations, which results in complex non-smooth
search spaces with multiple local minima. Finding solutions approaching global
optimum may present a computationally intractable task. Here, we develop a
framework that allows expediting the search of solutions close to global
optimum on complex optimization spaces. We study the way representative black
box optimization algorithms work, including genetic algorithm (GA), particle
swarm optimization (PSO), simulated annealing (SA), and mesh adaptive direct
search (NOMAD). We then propose and utilize a two-step approach that identifies
best performance algorithms on arbitrarily complex search spaces. We reveal a
connection between the search space complexity and algorithm performance and
find that PSO and NOMAD consistently deliver better performance for mixed
integer problems encountered in photonic inverse design, particularly with the
account of material combinations. Our results differ from a commonly
anticipated advantage of GA. Our findings will foster more efficient design of
photonic systems with optimal performance.
