  Recent advances in automatic evaluation metrics for text have shown that deep
contextualized word representations, such as those generated by BERT encoders,
are helpful for designing metrics that correlate well with human judgements. At
the same time, it has been argued that contextualized word representations
exhibit sub-optimal statistical properties for encoding the true similarity
between words or sentences. In this paper, we present two techniques for
improving encoding representations for similarity metrics: a batch-mean
centering strategy that improves statistical properties; and a computationally
efficient tempered Word Mover Distance, for better fusion of the information in
the contextualized word representations. We conduct numerical experiments that
demonstrate the robustness of our techniques, reporting results over various
BERT-backbone learned metrics and achieving state of the art correlation with
human ratings on several benchmarks.
