  A class of dark photon dark matter models with ultralight masses would lead
to oscillation of a test body through a coupling with baryons or $B-L$ charge.
This periodical oscillation of an observer results in swing of a star's
apparent position due to the effect of aberration of light, which could be
probed with high-precision astrometry observations of stars in the Milky Way.
We propose to use the observations of stellar positions of a number of stars by
Gaia to search for this kind of dark photon dark matter. We show that this
astrometry method is able to give promising sensitivities to search for the
dark photon dark matter in the mass range of $10^{-23}\sim10^{-21}$ eV.
