  We study the coupled-top model with three large spins located on a triangle.
Depending on the coupling strength, there exist three phases: disordered
paramagnetic phase, ferromagnetic phase, and frustrated antiferromagnetic
phase, which can be distinguished by the mean-field approach. The
paramagnetic-ferromagnetic phase transition is accompanied by the breaking of
the global $Z_2$ symmetry, whereas the paramagnetic-antiferromagnetic phase
transition is accompanied by the breaking of both the global $Z_2$ symmetry and
the translational symmetry. Exact analytical results of higher-order quantum
effects beyond the mean-field contribution, such as the excitation energy,
quantum fluctuation, and von Neumann entropy, can be achieved by the
Holstein-Primakoff transformation and symplectic transformation in the
thermodynamic limit. Near the quantum critical point, the energy gap closes,
along with the divergence of the quantum fluctuation in certain quadrature and
von Neumann entropy. Particular attention should be paid to the
antiferromagnetic phase, where geometric frustration takes effect. The critical
behaviors in the antiferromagnetic phase are quite different from those in the
paramagnetic and ferromagnetic phases, which highlight the importance of
geometric frustration. The triangular coupled-top model provides a simple and
feasible platform to study the quantum phase transition and the novel critical
behaviors induced by geometric frustration.
