  Unconventional superconductivity has been suggested to be present at the
interface between bismuth and nickel in thin-film bilayers. In this work, we
study the structural, magnetic and superconducting properties of sputter
deposited Bi/Ni bilayers. As-grown, our films do not display a superconducting
transition, however, when stored at room temperature, after about 14 days our
bilayers develop a superconducting transition up to 3.8 K. To systematically
study the effect of low temperature annealing on our bilayers, we perform
structural characterization with X-ray diffraction and polarized neutron
reflectometry, along with magnetometry and low temperature electrical transport
measurements on samples annealed at $70\,^\circ$C. We show that the onset of
superconductivity in our samples is coincident with the formation of ordered
NiBi$_3$ intermetallic alloy, a known $s$-wave superconductor. We calculate
that the annealing process has an activation energy of $(0.86\pm 0.06)$eV. As a
consequence, gentle heating of the bilayers will cause formation of the
superconducting NiBi$_3$ at the Ni/Bi interface, which poses a challenge to
studying any distinct properties of Bi/Ni bilayers without degrading that
interface.
