  The bulk Rashba semiconductors BiTeX (X=I, Cl and Br) with intrinsically
enhanced Rashba spin-orbit coupling provide a new platform for investigation of
spintronic and magnetic phenomena in materials. We theoretically investigate
the interlayer exchange interaction between two ferromagnets deposited on
opposite surfaces of a bulk Rashba semiconductor BiTeI in its trivial and
topological insulator phases. In the trivial phase BiTeI, we find that for
ferromagnets with a magnetization orthogonal to the interface, the exchange
coupling is reminiscent of that of a conventional three-dimensional metal.
Remarkably, ferromagnets with a magnetization parallel to the interface display
a magnetic exchange qualitatively different from that of conventional
three-dimensional metal due to the spin-orbit coupling. In this case, the
interlayer exchange interaction acquires two periods of oscillations and decays
as the inverse of the thickness of the BiTeI layer. For topological BiTeI, the
magnetic exchange interaction becomes mediated only by the helical surface
states and acts between the one-dimensional spin chains at the edges of the
sample. The surface state-mediated interlayer exchange interaction allows for
the coupling of ferromagnets with non-collinear magnetization and displays a
decay power different from that of trivial BiTeI, allowing the detection of the
topological phase transition in this material. Our work provides insights into
the magnetic properties of these newly discovered materials and their possible
functionalization.
