  Recent work has shown that prompting language models with code-like
representations of natural language leads to performance improvements on
structured reasoning tasks. However, such tasks comprise only a small subset of
all natural language tasks. In our work, we seek to answer whether or not
code-prompting is the preferred way of interacting with language models in
general. We compare code and text prompts across three popular GPT models
(davinci, code-davinci-002, and text-davinci-002) on a broader selection of
tasks (e.g., QA, sentiment, summarization) and find that with few exceptions,
code prompts do not consistently outperform text prompts. Furthermore, we show
that the style of code prompt has a large effect on performance for some but
not all tasks and that fine-tuning on text instructions leads to better
relative performance of code prompts.
