  Neural style transfer (NST) is a deep learning technique that produces an
unprecedentedly rich style transfer from a style image to a content image. It
is particularly impressive when it comes to transferring style from a painting
to an image. NST was originally achieved by solving an optimization problem to
match the global statistics of the style image while preserving the local
geometric features of the content image. The two main drawbacks of this
original approach is that it is computationally expensive and that the
resolution of the output images is limited by high GPU memory requirements.
Many solutions have been proposed to both accelerate NST and produce images
with larger size. However, our investigation shows that these accelerated
methods all compromise the quality of the produced images in the context of
painting style transfer. Indeed, transferring the style of a painting is a
complex task involving features at different scales, from the color palette and
compositional style to the fine brushstrokes and texture of the canvas. This
paper provides a solution to solve the original global optimization for
ultra-high resolution (UHR) images, enabling multiscale NST at unprecedented
image sizes. This is achieved by spatially localizing the computation of each
forward and backward passes through the VGG network. Extensive qualitative and
quantitative comparisons, as well as a \textcolor{coverletter}{perceptual
study}, show that our method produces style transfer of unmatched quality for
such high-resolution painting styles. By a careful comparison, we show that
state-of-the-art fast methods are still prone to artifacts, thus suggesting
that fast painting style transfer remains an open problem. Source code is
available at https://github.com/bgalerne/scaling_painting_style_transfer.
