  User generated content (UGC) refers to videos that are uploaded by users and
shared over the Internet. UGC may have low quality due to noise and previous
compression. When re-encoding UGC for streaming or downloading, a traditional
video coding pipeline will perform rate-distortion (RD) optimization to choose
coding parameters. However, in the UGC video coding case, since the input is
not pristine, quality ``saturation'' (or even degradation) can be observed,
i.e., increased bitrate only leads to improved representation of coding
artifacts and noise present in the UGC input. In this paper, we study the
saturation problem in UGC compression, where the goal is to identify and avoid
during encoding, the coding parameters and rates that lead to quality
saturation. We proposed a geometric criterion for saturation detection that
works with rate-distortion optimization, and only requires a few frames from
the UGC video. In addition, we show how to combine the proposed saturation
detection method with existing video coding systems that implement
rate-distortion optimization for efficient compression of UGC videos.
