  In this work we study symmetry fractionalization of vison excitations in
topological $\mathbb{Z}_2$ spin liquids. We show that in the presence of the
full $\mathrm{SO}(3)$ spin rotational symmetry and if there is an odd number of
spin-$\frac12$ per unit cell, the symmetry fractionalization of visons is
completely fixed. On the other hand, visons can have different classes of
symmetry fractionalization if the spin rotational symmetry is reduced. As a
concrete example, we show that visons in the Balents-Fisher-Girvin
$\mathbb{Z}_2$ spin liquid have crystal symmetry fractionalization classes
which are not allowed in $\mathrm{SO}(3)$ symmetric spin liquids, due to the
reduced spin rotational symmetry.
