  The optimal zero delay coding of a finite state Markov source is considered.
The existence and structure of optimal codes are studied using a stochastic
control formulation. Prior results in the literature established the optimality
of deterministic Markov (Walrand-Varaiya type) coding policies for finite time
horizon problem, and the optimality of both deterministic nonstationary and
randomized stationary policies for the infinite time horizon problem. Our main
result here shows that for any irreducible and aperiodic Markov source with a
finite alphabet, \emph{deterministic and stationary} Markov coding policies are
optimal for the infinite horizon problem. In addition, the finite blocklength
(time horizon) performance on an optimal (stationary and Markov) coding policy
is shown to approach the infinite time horizon optimum at a rate $O(1/T)$. The
results are extended to systems where zero delay communication takes place
across a noisy channel with noiseless feedback.
