  Deep neural networks work well at approximating complicated functions when
provided with data and trained by gradient descent methods. At the same time,
there is a vast amount of existing functions that programmatically solve
different tasks in a precise manner eliminating the need for training. In many
cases, it is possible to decompose a task to a series of functions, of which
for some we may prefer to use a neural network to learn the functionality,
while for others the preferred method would be to use existing black-box
functions. We propose a method for end-to-end training of a base neural network
that integrates calls to existing black-box functions. We do so by
approximating the black-box functionality with a differentiable neural network
in a way that drives the base network to comply with the black-box function
interface during the end-to-end optimization process. At inference time, we
replace the differentiable estimator with its external black-box
non-differentiable counterpart such that the base network output matches the
input arguments of the black-box function. Using this "Estimate and Replace"
paradigm, we train a neural network, end to end, to compute the input to
black-box functionality while eliminating the need for intermediate labels. We
show that by leveraging the existing precise black-box function during
inference, the integrated model generalizes better than a fully differentiable
model, and learns more efficiently compared to RL-based methods.
