  Recent works have shown that attaching prompts to the input is effective at
conditioning Language Models (LM) to perform specific tasks. However, prompts
are always included in the input text during inference, thus incurring
substantial computational and memory overhead. Also, there is currently no
straightforward method of utilizing prompts that are longer than the maximum
input length of the LMs without incurring additional costs during inference. We
propose Prompt Injection (PI), a novel formulation of injecting the prompt into
the parameters of an LM to be an efficient alternative to attaching fixed
prompts to the input. We show that in scenarios with long fixed prompts, PI can
be up to 280 times more efficient in terms of total FLOPs than previous
approaches. We further explore methodologies for PI and show promising results
in persona-dependent conversation, semantic parsing, and zero-shot learning
with task instructions. Through these explorations, we show that PI can be a
promising direction for conditioning language models, especially in scenarios
with long and fixed prompts.
