  We develop an unrestricted Hartree-Fock mean-field method including Coulomb
repulsion $U$, $V$ and spin-orbital coupling $\lambda$ self-consistently to
investigate the mechanism of structural instability and magnetic ordering in
Ba$_2$MgReO$_6$. A comprehensive quadrupole phase diagram versus $U$ and $V$
with $\lambda$=0.28eV is calculated. Our results demonstrate that the
easy-plane anisotropy and the intersite Coulomb repulsion $V$ must be
considered to remove the orbital frustration. The increasing of $V$ to $>$20meV
would arrange quadrupole $Q_{x^2-y^2}$ antiparallelly, accompanied with small
parallel $Q_{3z^2-r^2}$, and stabilize Ba$_2$MgReO$_6$ into the body-centered
tetragonal structure. Such antiparallel $Q_{x^2-y^2}$ provides a new mechanism
of Dzyaloshinskii-Moriya interaction, and gives rise to the canted
antiferromagnetic (CAF) state along [110] axis. Moreover, sizable octupoles
such as $O_{21}^{31}$, $O_{21}^{33}$, $O_{21}^{34}$ and $O_{21}^{36}$ are
discovered for the first time in CAF state. Our study not only provides a
comprehensive understanding of the experiment results in Ba$_2$MgReO$_6$, but
also reveals some commonalities of 5d compounds.
