  We consider an analogue of Witten's $SL(2,\mathbb{Z})$ action on
three-dimensional QFTs with $U(1)$ symmetry for $2k$-dimensional QFTs with
$\mathbb{Z}_2$ $(k-1)$-form symmetry. We show that the $SL(2,\mathbb{Z})$
action only closes up to a multiplication by an invertible topological phase
whose partition function is the Brown-Kervaire invariant of the spacetime
manifold. We interpret it as part of the $SL(2,\mathbb{Z})$ anomaly of the bulk
$(2k+1)$-dimensional $\mathbb{Z}_2$ gauge theory.
