  Summarization systems make numerous "decisions" about summary properties
during inference, e.g. degree of copying, specificity and length of outputs,
etc. However, these are implicitly encoded within model parameters and specific
styles cannot be enforced. To address this, we introduce HydraSum, a new
summarization architecture that extends the single decoder framework of current
models to a mixture-of-experts version with multiple decoders. We show that
HydraSum's multiple decoders automatically learn contrasting summary styles
when trained under the standard training objective without any extra
supervision. Through experiments on three summarization datasets (CNN, Newsroom
and XSum), we show that HydraSum provides a simple mechanism to obtain
stylistically-diverse summaries by sampling from either individual decoders or
their mixtures, outperforming baseline models. Finally, we demonstrate that a
small modification to the gating strategy during training can enforce an even
stricter style partitioning, e.g. high- vs low-abstractiveness or high- vs
low-specificity, allowing users to sample from a larger area in the generation
space and vary summary styles along multiple dimensions.
