  Although text style transfer has witnessed rapid development in recent years,
there is as yet no established standard for evaluation, which is performed
using several automatic metrics, lacking the possibility of always resorting to
human judgement. We focus on the task of formality transfer, and on the three
aspects that are usually evaluated: style strength, content preservation, and
fluency. To cast light on how such aspects are assessed by common and new
metrics, we run a human-based evaluation and perform a rich correlation
analysis. We are then able to offer some recommendations on the use of such
metrics in formality transfer, also with an eye to their generalisability (or
not) to related tasks.
