  Keyphrase generation aims at generating topical phrases from a given text
either by copying from the original text (present keyphrases) or by producing
new keyphrases (absent keyphrases) that capture the semantic meaning of the
text. Encoder-decoder models are most widely used for this task because of
their capabilities for absent keyphrase generation. However, there has been
little to no analysis on the performance and behavior of such models for
keyphrase generation. In this paper, we study various tendencies exhibited by
three strong models: T5 (based on a pre-trained transformer),
CatSeq-Transformer (a non-pretrained Transformer), and ExHiRD (based on a
recurrent neural network). We analyze prediction confidence scores, model
calibration, and the effect of token position on keyphrases generation.
Moreover, we motivate and propose a novel metric framework, SoftKeyScore, to
evaluate the similarity between two sets of keyphrases by using softscores to
account for partial matching and semantic similarity. We find that SoftKeyScore
is more suitable than the standard F1 metric for evaluating two sets of given
keyphrases.
