  The two-dimensional Hubbard model defined for topological band structures
exhibiting a quantum spin Hall effect poses fundamental challenges in terms of
phenomenological characterization and microscopic classification. In the limit
of infinite coupling U at half filling, the spin model Hamiltonians resulting
from a strong coupling expansion show various forms of magnetic ordering
phenomena depending on the underlying spin-orbit coupling terms. We investigate
the infinite U limit of the Kane-Mele Hubbard model with z-axis intrinsic
spin-orbit coupling as well as its generalization to a generically
multi-directional spin orbit term which has been claimed to account for the
physical scenario in monolayer Na2IrO3. We find that the axial spin symmetry
which is kept in the former but broken in the latter has a fundamental impact
on the magnetic phase diagram as we vary the spin orbit coupling strength.
While the Kane-Mele spin model shows a continuous evolution from conventional
honeycomb Neel to XY antiferromagnetism which avoids the frustration imposed by
the increased spin-orbit coupling, the multi-directional spin-orbit term
induces a commensurate to incommensurate transition at intermediate coupling
strength, and yields a complex spiral state with a 72 site unit cell in the
limit of infinite spin-orbit coupling. From our findings, we conjecture that in
the case of broken axial spin symmetry there is a large propensity for an
additional phase at sufficiently large spin-orbit coupling and intermediate U.
