  We demonstrate that the sensitivity of high-precision pulsar timing
experiments will be ultimately limited by the broadband intensity modulation
that is intrinsic to the pulsar's stochastic radio signal. That is, as the peak
flux of the pulsar approaches that of the system equivalent flux density,
neither greater antenna gain nor increased instrumental bandwidth will improve
timing precision. These conclusions proceed from an analysis of the covariance
matrix used to characterise residual pulse profile fluctuations following the
template matching procedure for arrival time estimation. We perform such an
analysis on 25 hours of high-precision timing observations of the closest and
brightest millisecond pulsar, PSR J0437-4715. In these data, the standard
deviation of the post-fit arrival time residuals is approximately four times
greater than that predicted by considering the system equivalent flux density,
mean pulsar flux and the effective width of the pulsed emission. We develop a
technique based on principal component analysis to mitigate the effects of
shape variations on arrival time estimation and demonstrate its validity using
a number of illustrative simulations. When applied to our observations, the
method reduces arrival time residual noise by approximately 20%. We conclude
that, owing primarily to the intrinsic variability of the radio emission from
PSR J0437-4715 at 20 cm, timing precision in this observing band better than 30
- 40 ns in one hour is highly unlikely, regardless of future improvements in
antenna gain or instrumental bandwidth. We describe the intrinsic variability
of the pulsar signal as stochastic wideband impulse modulated self-noise
(SWIMS) and argue that SWIMS will likely limit the timing precision of every
millisecond pulsar currently observed by Pulsar Timing Array projects as larger
and more sensitive antennae are built in the coming decades.
