  Based on a generalized one-band Hubbard model, we study magnetic properties
of Mott insulating states for ultracold spin-3/2 fermionic atoms in optical
lattices. When the \textit{s}-wave scattering lengths for the total spin
$S=2,0$ satisfy conditions $a_{2}>a_{0}>0$, we apply a functional integral
approach to the half filled case, where the spin-quadrupole fluctuations
dominate. On a 2D square lattice, the saddle point solution yields a staggered
spin-quadrupole ordering at zero temperature with symmetry breaking from SO(5)
to SO(4). Both spin and spin-quadrupole static structure factors are
calculated, displaying highly anisotropic spin antiferromagnetic fluctuations
and antiferroquadrupole long-range correlations, respectively. When Gaussian
fluctuations around the saddle point are taken into account, spin-quadrupole
density waves with a linear dispersion are derived. Compared with the spin
density waves in the half filled spin-1/2 Hubbard model, the quadrupole density
wave velocity is saturated in the strong-coupling limit, and there are no
transverse spin-quadrupole mode couplings, as required by the SO(4) invariance
of the effective action. Finally, in the strong-coupling limit of the model
Hamiltonian, we derive the effective hyperfine spin-exchange interactions for
the Mott insulating phases in the quarter filled and half filled cases,
respectively.
