  The structural and magnetic properties of polycrystalline YMn1-xMxO3 (M = Ti,
Fe, Ga; x = 0, 0.1) have been studied by neutron powder diffraction and
magnetic measurements to probe the effect of Mn site doping on the frustration
behavior and magnetic structure of these compounds. The compounds are
isostructural and crystallize with hexagonal structure in P63cm space group. We
find that doping with these three ions, Ti4+ (d0), Fe3+ (d5) and Ga3+ (d10),
influences both the TN and magnetic structure, unlike other Mn-site dopants
reported previously. The magnetic structure of YMnO3 is described by
considering a linear combination of irreducible representations {\Gamma}3 and
{\Gamma}4 below TN ~ 75K and with decrease in temperature the ratio of
{\Gamma}3 and {\Gamma}4 changes. The mixing ratio of these two irreducible
representations remains constant on lowering of temperature in the Ga doped
compounds. The magnetic structure is modified on doping with nonmagnetic ion
Ti4+ (d0). It is described by the basis vectors of the irreducible
representation {\Gamma}2 with moment 2.3{\mu}B at 6 K. On doping with Fe3+ (d5)
the magnetic structure immediately below TN is explained by considering the
{\Gamma}3 irreducible representation. On further lowering of temperature, a
spin reorientation at ~ 35 K is observed. Below this temperature, the magnetic
structure of YMn0.9Fe0.1O3 is explained by considering {\Gamma}3 representation
with 51% mixing of {\Gamma}4. The ordered moments are found to be reduced from
the expected value for a Mn3+ ion in all these compounds indicating the
frustrated nature of these compounds. However, the frustration parameter, f is
significantly reduced in the case of Ti doped compound with {\Gamma}2
representation.
