  We present a comprehensive analysis of the $\gamma W$ interference radiative
correction to the neutron $\beta$-decay matrix element. Within a dispersion
relations approach, we compute the axial-vector part of the $\gamma W$ box
amplitude $\Box^{\gamma W}_{A}$ in terms of the isoscalar part of the
$F_3^{\gamma W}$ interference structure function. Using the latest available
phenomenology for $F_3^{\gamma W}$ from the nucleon elastic, resonance,
deep-inelastic, and Regge regions, we find the real part of the box correction
to be $\Box^{\gamma W}_A = 3.90(9) \times 10^{-3}$. This improved correction
gives a theoretical estimate of the CKM matrix element
$|V_{ud}|^2=0.94805(26)$, which represents a 4$\sigma$ violation of unitarity.
