  We report experimental studies of a series of BaFe$_2$S$_{3-x}$Se$_x$ ($0\leq
x\leq3$) single crystals and powder specimens using x-ray diffraction, neutron
diffraction, muon spin relaxation, and electrical transport measurements. A
structural transformation from Cmcm (BaFe$_2$S$_3$) to Pnma (BaFe$_2$Se$_3$)
was identified around $x = 0.7\sim 1$. Neutron diffraction measurements on the
samples with $x$ = 0.2, 0.4, and 0.7 reveal that the N$\'{e}$el temperature of
the stripe antiferromagnetic order is gradually suppressed from $\sim$120 to 85
K, while the magnitude of the ordered Fe$^{2+}$ moments shows very little
variation. Similarly, the block antiferromagnetic order in BaFe$_2$Se$_3$
remains robust for $1.5\leq x\leq3$ with negligible variation in the ordered
moment and a slight decrease of the N$\'{e}$el temperature from 250 K ($x=3$)
to 225 K ($x=1.5$). The sample with $x=1$ near the Cmcm and Pnma border shows
coexisting, two-dimensional, short-range stripe- and block-type
antiferromagnetic correlations. The system remains insulating for all $x$, but
the thermal activation gap shows an abrupt increase when traversing the
boundary from the Cmcm stripe phase to the Pnma block phase. The results
demonstrate that the crystal structure, magnetic order, and electronic
properties are strongly coupled in the BaFe$_2$S$_{3-x}$Se$_x$ system.
