  The quantum spin liquid is an enigmatic quantum state in insulating magnets,
in which conventional long-range order is suppressed by strong quantum
fluctuations. Recently, an unconventional phase transition was reported between
the low-temperature quantum spin liquid and the high-temperature paramagnet in
the Kitaev model on a three-dimensional hyperhoneycomb lattice. Here, we show
that a similar "liquid-gas" transition takes place in another three-dimensional
lattice, the hyperoctagon lattice. We investigate the critical phenomena by
adopting the Green-function based Monte Carlo technique with the kernel
polynomial method, which enables systematic analysis of up to 2048 sites. The
critical temperature is lower than that in the hyperhoneycomb case, reflecting
the smaller flux gap. We also discuss the transition on the basis of an
effective model in the anisotropic limit.
