  A complete mathematical theory of nonequilibrium thermodynamics of stochastic
systems in terms of master equations is presented. As generalizations of
isothermal entropy and free energy, two functions of states play central roles:
the Gibbs entropy $S$ and the relative entropy $F$, which are related via the
stationary distribution of the stochastic dynamics. $S$ satisfies the
fundamental entropy balance equation $dS/dt=e_p-h_d/T$ with entropy production
rate $e_p\ge 0$ and heat dissipation rate $h_d$, while $dF/dt=-f_d\le 0$. For
closed systems that satisfy detailed balance: $Te_p(t)=f_d(t)$. For open system
one has $Te_p(t)=f_d(t)+Q_{hk}(t)$ where the housekeeping heat $Q_{hk}\ge 0$
was first introduced in the phenomenological nonequilibrium steady state
thermodynamics. Entropy production $e_p$ consists of free energy dissipation
associated with spontaneous relaxation, $f_d$, and active energy pumping that
sustains the open system $Q_{hk}$. The amount of excess heat involved in the
relaxation $Q_{ex}=h_d-Q_{hk} = f_d-T(dS/dt)$.
