  X-ray absorption of $\gamma$-ray burst (GRB) afterglows is prevalent yet
poorly understood. X-ray derived neutral hydrogen column densities ($N_{\rm
H}$) of GRB X-ray afterglows show an increase with redshift, which might give a
clue for the origin of this absorption. We use more than 350 X-ray afterglows
with spectroscopic redshift ($z$) from the Swift XRT repository as well as over
100 Ly\,$\alpha$ absorption measurements in $z>1.6$ sources. The observed trend
of the average optical depth $\tau$ at 0.5 keV is consistent with both a sharp
increase of host $N_{\rm H}(z)$, and an absorbing diffuse intergalactic medium,
along with decreasing host contribution to $\tau$. We analyze a sub-sample of
high-$z$ GRBs with $N_{\rm H}$ derived both from the X-ray afterglow and the
Ly\,$\alpha$ line. The increase of X-ray derived $N_{\rm H}(z)$ is contrasted
by no such increase in the Ly\,$\alpha$ derived column density. We argue that
this discrepancy implies a lack of association between the X-ray and
Ly\,$\alpha$ absorbers at high-$z$. This points towards the X-ray absorption at
high $z$ being dominated by an intervening absorber, which lends credibility to
an absorbing intergalactic medium contribution.
