  Despite the recent upsurge of activity in image-based non-photorealistic
rendering (NPR), and in particular portrait image stylisation, due to the
advent of neural style transfer, the state of performance evaluation in this
field is limited, especially compared to the norms in the computer vision and
machine learning communities. Unfortunately, the task of evaluating image
stylisation is thus far not well defined, since it involves subjective,
perceptual and aesthetic aspects. To make progress towards a solution, this
paper proposes a new structured, three level, benchmark dataset for the
evaluation of stylised portrait images. Rigorous criteria were used for its
construction, and its consistency was validated by user studies. Moreover, a
new methodology has been developed for evaluating portrait stylisation
algorithms, which makes use of the different benchmark levels as well as
annotations provided by user studies regarding the characteristics of the
faces. We perform evaluation for a wide variety of image stylisation methods
(both portrait-specific and general purpose, and also both traditional NPR
approaches and neural style transfer) using the new benchmark dataset.
