  Magnetic skyrmions, vortex-like topological spin textures, have attracted
much interest in a wide range of research fields from fundamental physics to
spintronics applications. Recently, growing attention has also been paid to
antiskyrmions emerging in opposite topological charge in non-centrosymmetric
magnets with $D_{2\mathrm{d}}$ or $S_4$ symmetry. In these magnets, complex
interplay among anisotropic Dzyaloshinskii-Moriya interaction, uniaxial
magnetic anisotropy, and magnetic dipolar interactions generates a variety of
magnetic structures. However, the relation between the stability of
antiskyrmions and these magnetic interactions remains to be elucidated. In this
work, we control the uniaxial magnetic anisotropy of schreibersite (Fe,Ni)$_3$P
with $S_4$ symmetry by doping and investigate its impact on the stability of
antiskyrmions. Our magnetometry study, supported by ferromagnetic resonance
spectroscopy, shows that the variation of the Ni content and slight doping with
4$d$ transition metals considerably change the magnetic anisotropy. In
particular, doping with Pd induces easy-axis anisotropy, giving rise to
formation of antiskyrmions, while a temperature-induced spin reorientation is
observed in a Rh-doped compound. In combination with Lorentz transmission
electron microscopy and micromagnetic simulations, we quantitatively analyze
the stability of antiskyrmion as functions of uniaxial anisotropy and
demagnetization energy, and demonstrate that subtle balance between them is
necessary to stabilize the antiskyrmions.
