  Many computational tasks can be naturally expressed as a composition of a DNN
followed by a program written in a traditional programming language or an API
call to an LLM. We call such composites "neural programs" and focus on the
problem of learning the DNN parameters when the training data consist of
end-to-end input-output labels for the composite. When the program is written
in a differentiable logic programming language, techniques from neurosymbolic
learning are applicable, but in general, the learning for neural programs
requires estimating the gradients of black-box components. We present an
algorithm for learning neural programs, called ISED, that only relies on
input-output samples of black-box components. For evaluation, we introduce new
benchmarks that involve calls to modern LLMs such as GPT-4 and also consider
benchmarks from the neurosymolic learning literature. Our evaluation shows that
for the latter benchmarks, ISED has comparable performance to state-of-the-art
neurosymbolic frameworks. For the former, we use adaptations of prior work on
gradient approximations of black-box components as a baseline, and show that
ISED achieves comparable accuracy but in a more data- and sample-efficient
manner.
