  For summarization, human preference is critical to tame outputs of the
summarizer in favor of human interests, as ground-truth summaries are scarce
and ambiguous. Practical settings require dynamic exchanges between human and
AI agent wherein feedback is provided in an online manner, a few at a time. In
this paper, we introduce a new framework to train summarization models with
preference feedback interactively. By properly leveraging offline data and a
novel reward model, we improve the performance regarding ROUGE scores and
sample-efficiency. Our experiments on three various datasets confirm the
benefit of the proposed framework in active, few-shot and online settings of
preference learning.
