  The series of intermetallic compounds $R$NiSi$_3$ ($R$ = rare earth) shows
interesting magnetic properties evolving with $R$ and metamagnetic transitions
under applied magnetic field for some of the compounds. The microscopic
magnetic structures must be determined to rationalize such rich behavior. Here,
resonant x-ray magnetic diffraction experiments are performed on single
crystals of GdNiSi$_{3}$ and TbNiSi$_{3}$ at zero field. The primitive magnetic
unit cell matches the chemical cell below the N\'eel temperatures $T_{N}$ =
22.2 and 33.2 K, respectively. The magnetic structure is determined to be the
same for both compounds (magnetic space group $Cmmm'$). It features
ferromagnetic {\it ac} planes that are stacked in an antiferromagnetic $+-+-$
pattern, with the rare-earth magnetic moments pointing along the $\vec{a}$
direction, which contrasts with the $+--+$ stacking and moment direction along
the $\vec{b}$ axis previously reported for YbNiSi$_3$. This indicates a sign
reversal of the coupling constant between second-neighbor $R$ planes as $R$ is
varied from Gd and Tb to Yb. The long {\it b} lattice parameter of GdNiSi$_{3}$
and TbNiSi$_{3}$ shows a magnetoelastic expansion upon cooling below $T_N$,
pointing to the conclusion that the $+-+-$ stacking is stabilized under lattice
expansion. A competition between distinct magnetic stacking patterns with
similar exchange energies tuned by the size of $R$ sets the stage for the
magnetic ground state instability observed along this series.
