  Summarization systems are ultimately evaluated by human annotators and
raters. Usually, annotators and raters do not reflect the demographics of end
users, but are recruited through student populations or crowdsourcing platforms
with skewed demographics. For two different evaluation scenarios -- evaluation
against gold summaries and system output ratings -- we show that summary
evaluation is sensitive to protected attributes. This can severely bias system
development and evaluation, leading us to build models that cater for some
groups rather than others.
