  The possible variation of the electromagnetic fine structure constant
$\alpha_e$ on cosmological scales arouses great interests in recent years. The
strongly lensed gravitational waves and the electromagnetic counterparts could
be used to test this variation. Under the assumption that the speed of photon
could be modified, while the speed of GW is the same as GR predicated, and they
both propagate in a flat Friedman-Robertson-Walker universe, we investigate the
difference of time delays of the images and derive the upper bound of the
variation of $\alpha_e$. For a typical lensing system in the standard
cosmological models, we obtain $B\cos\theta\leq 1.85\times10^{-5}$, where $B$
is the dipolar amplitude and $\theta$ is the angle between observation and the
preferred direction. Our result is consistent with the most up-to-date
observations on $\alpha_e$. In addition, the observations of strongly lensed
gravitational waves and the electromagnetic counterparts could be used to test
which types of alternative theories of gravity can account for the variation of
$\alpha_e$.
