  On the basis of the multi-orbital dynamical mean field theory, a
three-orbital Hubbard model with a relativistic spin-orbit coupling (SOC) is
studied at five electrons per site. The numerical calculations are performed by
employing the continuous-time quantum Monte Carlo (CTQMC) method based on the
strong coupling expansion. We find that appropriately choosing bases, i.e., the
maximally spin-orbit-entangled bases, drastically improve the sign problem in
the CTQMC calculations, which enables us to treat exactly the full Hund's
coupling and pair hopping terms. This improvement is also essential to reach at
low temperatures for a large SOC region where the SOC most significantly
affects the electronic structure. We show that a metal-insulator transition is
induced by the SOC for fixed Coulomb interactions. The insulating state for
smaller Coulomb interactions is antiferromagnetically ordered with the local
effective total angular momentum $j=1/2$, in which the $j=1/2$ based band is
essentially half-filled while the $j=3/2$ based bands are completely occupied.
More interestingly, for larger Coulomb interactions, we find that an excitonic
insulating state emerges, where the condensation of an electron-hole pair in
the $j=1/2$ and $j=3/2$ based bands occurs. The origin of the excitonic
insulator as well as the experimental implication is discussed.
