  We study the $2+1$ dimensional boson-fermion duality in the presence of
background curvature and electromagnetic fields. The main players are, on the
one hand, a massive complex $|\phi|^4$ scalar field coupled to a $U(1)$
Maxwell-Chern-Simons gauge field at level $1$, representing a relativistic
composite boson with one unit of attached flux, and on the other hand, a
massive Dirac fermion. We show that, in a curved background and at the level of
the partition function, the relativistic composite boson, in the infinite
coupling limit, is dual to a short-range interacting Dirac fermion. The
coupling to the gravitational spin connection arises naturally from the spin
factors of the Wilson loop in the Chern-Simons theory. A non-minimal coupling
to the scalar curvature is included on the bosonic side in order to obtain
agreement between partition functions. Although an explicit Lagrangian
expression for the fermionic interactions is not obtained, their short-range
nature constrains them to be irrelevant, which protects the duality in its
strong interpretation as an exact mapping at the IR fixed point between a
Wilson-Fischer-Chern-Simons complex scalar and a free Dirac fermion. We also
show that, even away from the IR, keeping the $|\phi|^4$ term is of key
importance as it provides the short-range bosonic interactions necessary to
prevent intersections of worldlines in the path integral, thus forbidding
unknotting of knots and ensuring preservation of the worldline topologies.
