  Hallucination is a known issue for neural abstractive summarization models.
Recent work suggests that the degree of hallucination may depend on errors in
the training data. In this work, we propose a new method called Contrastive
Parameter Ensembling (CaPE) to use training data more effectively, utilizing
variations in noise in training samples to reduce hallucination. We first
select clean and noisy subsets from the training data using different automatic
factual metrics. Then, we fine-tune a base summarization model, which is
trained on all training samples, on the clean (noisy) subset to obtain an
\textit{expert} (\textit{anti-expert}) model. Finally, we adjust the parameters
of base model by the difference between parameters of the \textit{expert} and
\textit{anti-expert} models, steering the base model towards the
\textit{expert} model and away from the \textit{anti-expert} model.
Experimental results show that CaPE improves performance across different
automatic factual metrics and human evaluation, with the maximum improvement of
16.69\% and 15.78\% on summary-level dependency-arc entailment accuracy for the
XSUM and CNN/DM datasets. The improvement in factual performance does not
degrade the performance on other metrics of informativeness such as ROUGE.
