  The evaluation of recent embedding-based evaluation metrics for text
generation is primarily based on measuring their correlation with human
evaluations on standard benchmarks. However, these benchmarks are mostly from
similar domains to those used for pretraining word embeddings. This raises
concerns about the (lack of) generalization of embedding-based metrics to new
and noisy domains that contain a different vocabulary than the pretraining
data. In this paper, we examine the robustness of BERTScore, one of the most
popular embedding-based metrics for text generation. We show that (a) an
embedding-based metric that has the highest correlation with human evaluations
on a standard benchmark can have the lowest correlation if the amount of input
noise or unknown tokens increases, (b) taking embeddings from the first layer
of pretrained models improves the robustness of all metrics, and (c) the
highest robustness is achieved when using character-level embeddings, instead
of token-based embeddings, from the first layer of the pretrained model.
