  In recent years, deep perceptual loss has been widely and successfully used
to train machine learning models for many computer vision tasks, including
image synthesis, segmentation, and autoencoding. Deep perceptual loss is a type
of loss function for images that computes the error between two images as the
distance between deep features extracted from a neural network. Most
applications of the loss use pretrained networks called loss networks for deep
feature extraction. However, despite increasingly widespread use, the effects
of loss network implementation on the trained models have not been studied.
  This work rectifies this through a systematic evaluation of the effect of
different pretrained loss networks on four different application areas.
Specifically, the work evaluates 14 different pretrained architectures with
four different feature extraction layers. The evaluation reveals that VGG
networks without batch normalization have the best performance and that the
choice of feature extraction layer is at least as important as the choice of
architecture. The analysis also reveals that deep perceptual loss does not
adhere to the transfer learning conventions that better ImageNet accuracy
implies better downstream performance and that feature extraction from the
later layers provides better performance.
