  Recently, pretrained language models (PLMs) have had exceptional success in
language generation. To leverage the rich knowledge encoded by PLMs, a simple
yet powerful paradigm is to use prompts in the form of either discrete tokens
or continuous embeddings. In existing studies, these prompting methods are
typically independent of the inputs, lacking sufficient consideration of input
semantics. To address this issue, we propose a novel continuous prompting
approach, called context-tuning, to fine-tuning PLMs for natural language
generation. Firstly, the prompts are derived based on the input text to elicit
useful knowledge from PLMs for generation. We refer to such prompts as
contextualized prompts. Secondly, we use continuous inverse prompting to
improve the process of natural language generation by modeling an inverse
generation process from output to input, making the generated text more
relevant to the inputs. Furthermore, we utilize a lightweight context-tuning
method that fine-tunes only 0.12% of the parameters while maintaining good
performance. Our code is publicly available at
https://github.com/RUCAIBox/Context-Tuning.
