  We consider the semantic rate-distortion problem motivated by task-oriented
video compression. The semantic information corresponding to the task, which is
not observable to the encoder, shows impacts on the observations through a
joint probability distribution. The similarities among intra-frame segments and
inter-frames in video compression are formulated as side information available
at both the encoder and the decoder. The decoder is interested in recovering
the observation and making an inference of the semantic information under
certain distortion constraints. We establish the information-theoretic limits
for the tradeoff between compression rates and distortions by fully
characterizing the rate-distortion function. We further evaluate the
rate-distortion function under specific Markov conditions for three scenarios:
i) both the task and the observation are binary sources; ii) the task is a
binary classification of an integer observation as even and odd; iii) Gaussian
correlated task and observation. We also illustrate through numerical results
that recovering only the semantic information can reduce the coding rate
comparing to recovering the source observation.
