  Automated machine learning has gained a lot of attention recently. Building
and selecting the right machine learning models is often a multi-objective
optimization problem. General purpose machine learning software that
simultaneously supports multiple objectives and constraints is scant, though
the potential benefits are great. In this work, we present a framework called
Autotune that effectively handles multiple objectives and constraints that
arise in machine learning problems. Autotune is built on a suite of
derivative-free optimization methods, and utilizes multi-level parallelism in a
distributed computing environment for automatically training, scoring, and
selecting good models. Incorporation of multiple objectives and constraints in
the model exploration and selection process provides the flexibility needed to
satisfy trade-offs necessary in practical machine learning applications.
Experimental results from standard multi-objective optimization benchmark
problems show that Autotune is very efficient in capturing Pareto fronts. These
benchmark results also show how adding constraints can guide the search to more
promising regions of the solution space, ultimately producing more desirable
Pareto fronts. Results from two real-world case studies demonstrate the
effectiveness of the constrained multi-objective optimization capability
offered by Autotune.
