  The critical behavior of a model with N-vector complex order parameter and
three quartic coupling constants that describes phase transitions in
unconventional superconductors, helical magnets, stacked triangular
antiferromagnets, superfluid helium-3, and zero-temperature transitions in
fully frustrated Josephson-junction arrays is studied within the field-
theoretical renormalization-group approach in three dimensions. To obtain
qualitatively and quantitatively correct results perturbative expansions for
\beta-functions and critical exponents are calculated up to three-loop order
and resummed by means of the generalized Pade-Borel procedure. Fixed-point
coordinates, critical exponent values, RG flows, etc. are found for the
physically interesting cases N = 2 and N = 3. Marginal values of N at which the
topology of the flow diagram changes are determined as well. In most cases the
systems mentioned are shown to undergo fluctuation-driven first-order phase
transitions. Continuous transitions are allowed in hexagonal d-wave
superconductors, in planar helical magnets (into sinusoidal linearly-polarized
phase), and in triangular antiferromagnets (into simple unfrustrated ordered
states) with critical exponents \gamma = 1.336, \nu = 0.677, \alpha = -0.030,
\beta = 0.347, \eta = 0.026, which are hardly believed to be experimentally
distinguishable from those of the 3D XY model. The chiral fixed point of RG
equations is found to exist and possess some domain of attraction provided N >
3. Thus, magnets with Heisenberg (N = 3) and XY-like (N = 2) spins should not
demonstrate chiral critical behavior with unusual critical exponents; they can
approach the chiral state only via first-order phase transitions.
