  Relic gravitational waves (RGWs) generated in the early Universe form a
stochastic GW background, which can be directly probed by measuring the timing
residuals of millisecond pulsars. In this paper, we investigate the constraints
on the RGWs and on the inflationary parameters by the observations of current
and potential future pulsar timing arrays. In particular, we focus on effects
of various cosmic phase transitions (e.g. $e^{+}e^{-}$ annihilation, QCD
transition and SUSY breaking) and relativistic free-streaming gases (neutrinos
and dark fluids) in the general scenario of the early Universe, which have been
neglected in the previous works. We find that the phase transitions can
significantly damp the RGWs in the sensitive frequency range of pulsar timing
arrays, and the upper limits of tensor-to-scalar ratio $r$ increase by a factor
$\sim 2$ for both current and future observations. However, the effects of
free-steaming neutrinos and dark fluids are all too small to be detected.
Meanwhile, we find that, if the effective equation of state $w$ in the early
Universe is larger than $1/3$, i.e. deviating from the standard hot big bang
scenario, the detection of RGWs by pulsar timing arrays becomes much more
promising.
