  Work on summarization has explored both reinforcement learning (RL)
optimization using ROUGE as a reward and syntax-aware models, such as models
those input is enriched with part-of-speech (POS)-tags and dependency
information. However, it is not clear what is the respective impact of these
approaches beyond the standard ROUGE evaluation metric. Especially, RL-based
for summarization is becoming more and more popular. In this paper, we provide
a detailed comparison of these two approaches and of their combination along
several dimensions that relate to the perceived quality of the generated
summaries: number of repeated words, distribution of part-of-speech tags,
impact of sentence length, relevance and grammaticality. Using the standard
Gigaword sentence summarization task, we compare an RL self-critical sequence
training (SCST) method with syntax-aware models that leverage POS tags and
Dependency information. We show that on all qualitative evaluations, the
combined model gives the best results, but also that only training with RL and
without any syntactic information already gives nearly as good results as
syntax-aware models with less parameters and faster training convergence.
