  Anisotropic inflation is an interesting model with an U(1) gauge field and it
predicts the statistical anisotropy of the curvature perturbation characterized
by a parameter $g_*$. However, we find that the background gauge field does not
follow the classical attractor solution due to the stochastic effect. We
develop the stochastic formalism of a vector field and solve Langevin and
Fokker-Planck equations. It is shown that this model is excluded by the CMB
constraint $g_*\le 10^{-2}$ with a high probability about $99.999\%$.
