  Natural Language Processing (NLP) systems are increasingly taking the form of
multi-stage pipelines involving multiple distinct language models (LMs) and
prompting strategies. Here we address the question of how to fine-tune such
systems to improve their performance. We cast this as a problem of optimizing
the underlying LM weights and the prompting strategies together, and consider a
challenging but highly realistic scenario in which we have no gold labels for
any intermediate stages in the pipeline. To address this challenge, we evaluate
approximate optimization strategies in which we bootstrap training labels for
all pipeline stages and use these to optimize the pipeline's prompts and
fine-tune its weights alternatingly. In experiments with multi-hop QA,
mathematical reasoning, and feature-based classification, we find that simple
approaches for optimizing the prompts and weights together outperform directly
optimizing weights alone and prompts alone by up to 65% and 5%, respectively,
on average across LMs and tasks. We will release our new optimizers in DSPy at
http://dspy.ai
