  The Cosmic Microwave Background is characterized by temperature and linear
polarization fields. Dipole modulation in the temperature field has been
extensively studied in the context of hemispherical power asymmetry. In this
article, we show that a dipole modulation, and in general, any kind of
modulation isn't allowed in the $E$ and $B$ modes. This is the main result of
this paper. This result explains why no evidence of modulation in $E$ mode has
been found in the literature. On the contrary, the linear polarization fields
$Q$ and $U$ have no such restrictions. We show that modulation under certain
situations can be thought of as local $U(1)$ gauge transformations on the
surface of a sphere. As far as the modulation function is concerned, we show
that physical considerations enforce it to be (i) a spin $0$ field and (ii) a
scalar under parity. As masking is a specific type of modulation, our study
suggests that direct masking of $E$ mode isn't also possible. Masking in the
$E$ map can only be applied through $Q$ and $U$ fields. This means that in
principle, leaking of $E$ and $B$ mode powers into each other is unavoidable.
