  In modern video encoders, rate control is a critical component and has been
heavily engineered. It decides how many bits to spend to encode each frame, in
order to optimize the rate-distortion trade-off over all video frames. This is
a challenging constrained planning problem because of the complex dependency
among decisions for different video frames and the bitrate constraint defined
at the end of the episode.
  We formulate the rate control problem as a Partially Observable Markov
Decision Process (POMDP), and apply imitation learning to learn a neural rate
control policy. We demonstrate that by learning from optimal video encoding
trajectories obtained through evolution strategies, our learned policy achieves
better encoding efficiency and has minimal constraint violation. In addition to
imitating the optimal actions, we find that additional auxiliary losses, data
augmentation/refinement and inference-time policy improvements are critical for
learning a good rate control policy. We evaluate the learned policy against the
rate control policy in libvpx, a widely adopted open source VP9 codec library,
in the two-pass variable bitrate (VBR) mode. We show that over a diverse set of
real-world videos, our learned policy achieves 8.5% median bitrate reduction
without sacrificing video quality.
