  Most existing black-box optimization methods assume that all variables in the
system being optimized have equal cost and can change freely at each iteration.
However, in many real world systems, inputs are passed through a sequence of
different operations or modules, making variables in earlier stages of
processing more costly to update. Such structure imposes a cost on switching
variables in early parts of a data processing pipeline. In this work, we
propose a new algorithm for switch cost-aware optimization called Lazy Modular
Bayesian Optimization (LaMBO). This method efficiently identifies the global
optimum while minimizing cost through a passive change of variables in early
modules. The method is theoretical grounded and achieves vanishing regret when
augmented with switching cost. We apply LaMBO to multiple synthetic functions
and a three-stage image segmentation pipeline used in a neuroscience
application, where we obtain promising improvements over prevailing cost-aware
Bayesian optimization algorithms. Our results demonstrate that LaMBO is an
effective strategy for black-box optimization that is capable of minimizing
switching costs in modular systems.
