  We present the first systematic study of the filling constraints to realize a
`trivial' insulator symmetric under magnetic space group $\mathcal{M}$. The
filling $\nu$ must be an integer multiple of $m^{\mathcal{M}}$ to avoid
spontaneous symmetry breaking or fractionalization in gapped phases. We improve
the value of $m^{\mathcal{M}}$ in the literature and prove the tightness of the
constraint for the majority of magnetic space groups. The result may shed light
on the material search of exotic magnets with fractionalization.
