  Using neutron diffraction, we have studied the magnetic field effect on
charge structures in the charge-ordered multiferroic material LuFe$_2$O$_4$. An
external magnetic field is able to change the magnitude and correlation lengths
of the charge valence order even before the magnetic order sets in. This
affects the dielectric and ferroelectric properties of the material and induces
a giant magneto-electric effect. Our results suggest that the magneto-electric
coupling in LuFe$_2$O$_4$ is likely due to magnetic field effect on local
spins, in clear contrast to the case in most other known multiferroic systems
where the bulk magnetic order is important.
