  Human perception is at the core of lossy video compression and yet, it is
challenging to collect data that is sufficiently dense to drive compression. In
perceptual quality assessment, human feedback is typically collected as a
single scalar quality score indicating preference of one distorted video over
another. In reality, some videos may be better in some parts but not in others.
We propose an approach to collecting finer-grained feedback by asking users to
use an interactive tool to directly optimize for perceptual quality given a
fixed bitrate. To this end, we built a novel web-tool which allows users to
paint these spatio-temporal importance maps over videos. The tool allows for
interactive successive refinement: we iteratively re-encode the original video
according to the painted importance maps, while maintaining the same bitrate,
thus allowing the user to visually see the trade-off of assigning higher
importance to one spatio-temporal part of the video at the cost of others. We
use this tool to collect data in-the-wild (10 videos, 17 users) and utilize the
obtained importance maps in the context of x264 coding to demonstrate that the
tool can indeed be used to generate videos which, at the same bitrate, look
perceptually better through a subjective study - and are 1.9 times more likely
to be preferred by viewers. The code for the tool and dataset can be found at
https://github.com/jenyap/video-annotation-tool.git
