  Quantum phase transitions in the two-dimensional Kugel-Khomski model on a
square lattice are studied using the plaquette mean field theory and the
entanglement renormalization ansatz. When $3z^2-r^2$ orbitals are favored by
the crystal field and Hund's exchange is finite, both methods give a
noncollinear exotic magnetic order which consists of four sublattices with
mutually orthogonal nearest neighbor and antiferromagnetic second neighbor
spins. We derive effective frustrated spin model with second and third neighbor
spin interactions which stabilize this phase and follow from spin-orbital
quantum fluctuations involving spin singlets entangled with orbital
excitations.
