  Common state-of-the-art video codecs are optimized to deliver a low bitrate
by providing a certain quality for the final human observer, which is achieved
by rate-distortion optimization (RDO). But, with the steady improvement of
neural networks solving computer vision tasks, more and more multimedia data is
not observed by humans anymore, but directly analyzed by neural networks. In
this paper, we propose a standard-compliant feature-based RDO (FRDO) that is
designed to increase the coding performance, when the decoded frame is analyzed
by a neural network in a video coding for machine scenario. To that extent, we
replace the pixel-based distortion metrics in conventional RDO of VTM-8.0 with
distortion metrics calculated in the feature space created by the first layers
of a neural network. Throughout several tests with the segmentation network
Mask R-CNN and single images from the Cityscapes dataset, we compare the
proposed FRDO and its hybrid version HFRDO with different distortion measures
in the feature space against the conventional RDO. With HFRDO, up to 5.49 %
bitrate can be saved compared to the VTM-8.0 implementation in terms of
Bj{\o}ntegaard Delta Rate and using the weighted average precision as quality
metric. Additionally, allowing the encoder to vary the quantization parameter
results in coding gains for the proposed HFRDO of up 9.95 % compared to
conventional VTM.
