  The zero-temperature phase diagram of the $J_1-J_2$ SU($N$) antiferromagnetic
Heisenberg spin chain is investigated by means of complementary field theory
and numerical approaches for general $N$. A fully gapped SU($N$) valence bond
solid made of $N$ sites is formed above a critical value of $J_2/J_1$ for all
$N$. We find that the extension of this $N$-merized phase for larger values of
$J_2$ strongly depends on the parity of $N$. For even $N$, the phase smoothly
interpolates to the large $J_2$ regime where the model can be viewed as a
zigzag SU($N$) two-leg spin ladder. The phase exhibits both a $N$-merized
ground state and incommensurate spin-spin correlations. In stark contrast to
the even case, we show that the $N$-merized phase with odd $N$ only has a
finite extent with no incommensuration. A gapless phase in the SU($N$)$_1$
universality class is stabilized for larger $J_2$ that stems from the existence
of a massless renormalization group flow from SU($N$)$_2$ to SU($N$)$_1$
conformal field theories when $N$ is odd.
