  Current efficient fine-tuning methods (e.g., adapters, prefix-tuning, etc.)
have optimized conditional text generation via training a small set of extra
parameters of the neural language model, while freezing the rest for
efficiency. While showing strong performance on some generation tasks, they
don't generalize across all generation tasks. We show that soft-prompt based
conditional text generation can be improved with simple and efficient methods
that simulate modeling the discourse structure of human written text. We
investigate two design choices: First, we apply \textit{hierarchical blocking}
on the prefix parameters to simulate a higher-level discourse structure of
human written text. Second, we apply \textit{attention sparsity} on the prefix
parameters at different layers of the network and learn sparse transformations
on the softmax-function. We show that structured design of prefix parameters
yields more coherent, faithful and relevant generations than the baseline
prefix-tuning on all generation tasks.
