  We consider the problem of minimizing a composite convex function with two
different access methods: an oracle, for which we can evaluate the value and
gradient, and a structured function, which we access only by solving a convex
optimization problem. We are motivated by two associated technological
developments. For the oracle, systems like PyTorch or TensorFlow can
automatically and efficiently compute gradients, given a computation graph
description. For the structured function, systems like CVXPY accept a high
level domain specific language description of the problem, and automatically
translate it to a standard form for efficient solution. We develop a method
that makes minimal assumptions about the two functions, does not require the
tuning of algorithm parameters, and works well in practice across a variety of
problems. Our algorithm combines a number of well-known ideas, including a
low-rank quasi-Newton approximation of curvature, piecewise affine lower bounds
from bundle-type methods, and two types of damping to ensure stability. We
illustrate the method on stochastic optimization, utility maximization, and
risk-averse programming problems, showing that our method is more efficient
than standard solvers when the oracle function contains much data.
