  For many prediction tasks, stakeholders desire not only predictions but also
supporting evidence that a human can use to verify its correctness. However, in
practice, additional annotations marking supporting evidence may only be
available for a minority of training examples (if available at all). In this
paper, we propose new methods to combine few evidence annotations (strong
semi-supervision) with abundant document-level labels (weak supervision) for
the task of evidence extraction. Evaluating on two classification tasks that
feature evidence annotations, we find that our methods outperform baselines
adapted from the interpretability literature to our task. Our approach yields
substantial gains with as few as hundred evidence annotations. Code and
datasets to reproduce our work are available at
https://github.com/danishpruthi/evidence-extraction.
