  Critical properties of a geometrically frustrated generalized $XY$ model with
antiferromagnetic (AFM) and third-order antinematic (AN3) couplings on a
triangular lattice are studied by Monte Carlo simulation. It is found that such
a generalization leads to a phase diagram consisting of three different
quasi-long-range ordered (QLRO) phases. Compared to the model with the
second-order antinematic (AN2) coupling, besides the AFM and AN3 phases which
appear in the limits of relatively strong AFM and AN3 interactions,
respectively, it includes an additional complex canted antiferromagnetic (CAFM)
phase. It emerges at lower temperatures, wedged between the AFM and AN3 phases,
as a result of the competition between the AFM and AN3 couplings, which is
absent in the model with the AN2 coupling. The AFM-CAFM and AN3-CAFM phase
transitions are concluded to belong to the weak Ising and weak three-state
Potts universality classes, respectively. Additionally, all three QLRO phases
also feature true LRO of the standard and generalized chiralities, which both
vanish simultaneously at second-order phase transitions with non-Ising critical
exponents and the critical temperatures slightly higher than the magnetic and
nematic order-disorder transition temperatures.
