  Lack of factual correctness is an issue that still plagues state-of-the-art
summarization systems despite their impressive progress on generating seemingly
fluent summaries. In this paper, we show that factual inconsistency can be
caused by irrelevant parts of the input text, which act as confounders. To that
end, we leverage information-theoretic measures of causal effects to quantify
the amount of confounding and precisely quantify how they affect the
summarization performance. Based on insights derived from our theoretical
results, we design a simple multi-task model to control such confounding by
leveraging human-annotated relevant sentences when available. Crucially, we
give a principled characterization of data distributions where such confounding
can be large thereby necessitating the use of human annotated relevant
sentences to generate factual summaries. Our approach improves faithfulness
scores by 20\% over strong baselines on AnswerSumm
\citep{fabbri2021answersumm}, a conversation summarization dataset where lack
of faithfulness is a significant issue due to the subjective nature of the
task. Our best method achieves the highest faithfulness score while also
achieving state-of-the-art results on standard metrics like ROUGE and METEOR.
We corroborate these improvements through human evaluation.
