  Reshaping, a point operation that alters the characteristics of signals, has
been shown capable of improving the compression ratio in video coding
practices. Out-of-loop reshaping that directly modifies the input video signal
was first adopted as the supplemental enhancement information (SEI) for the
HEVC/H.265 without the need to alter the core design of the video codec.
VVC/H.266 further improves the coding efficiency by adopting in-loop reshaping
that modifies the residual signal being processed in the hybrid coding loop. In
this paper, we theoretically analyze the rate-distortion performance of the
in-loop reshaping and use experiments to verify the theoretical result. We
prove that the in-loop reshaping can improve coding efficiency when the entropy
coder adopted in the coding pipeline is suboptimal, which is in line with the
practical scenarios that video codecs operate in. We derive the PSNR gain in a
closed form and show that the theoretically predicted gain is consistent with
that measured from experiments using standard testing video sequences.
