  Despite recent progress in abstractive summarization, systems still suffer
from faithfulness errors. While prior work has proposed models that improve
faithfulness, it is unclear whether the improvement comes from an increased
level of extractiveness of the model outputs as one naive way to improve
faithfulness is to make summarization models more extractive. In this work, we
present a framework for evaluating the effective faithfulness of summarization
systems, by generating a faithfulnessabstractiveness trade-off curve that
serves as a control at different operating points on the abstractiveness
spectrum. We then show that the Maximum Likelihood Estimation (MLE) baseline as
well as a recently proposed method for improving faithfulness, are both worse
than the control at the same level of abstractiveness. Finally, we learn a
selector to identify the most faithful and abstractive summary for a given
document, and show that this system can attain higher faithfulness scores in
human evaluations while being more abstractive than the baseline system on two
datasets. Moreover, we show that our system is able to achieve a better
faithfulness-abstractiveness trade-off than the control at the same level of
abstractiveness.
