  Recently several studies have pointed out that gravitational-wave detectors
are sensitive to ultralight vector dark matter and can improve the current best
constraints given by the Equivalence Principle tests. While a
gravitational-wave detector is a highly precise measuring tool of the length
difference of its arms, its sensitivity is limited because the displacements of
its test mass mirrors caused by vector dark matter are almost common. In this
Letter we point out that the sensitivity is significantly improved if the
effect of finite light-traveling time in the detector's arms is taken into
account. This effect enables advanced LIGO to improve the constraints on the
$U(1)_{B-L}$ gauge coupling by an order of magnitude compared with the current
best constraints. It also makes the sensitivities of the future
gravitational-wave detectors overwhelmingly better than the current ones. The
factor by which the constraints are improved due to the new effect depends on
the mass of the vector dark matter, and the maximum improvement factors are
$470$, $880$, $1600$, $180$ and $1400$ for advanced LIGO, Einstein Telescope,
Cosmic Explorer, DECIGO and LISA respectively. Including the new effect, we
update the constraints given by the first observing run of advanced LIGO and
improve the constraints on the $U(1)_B$ gauge coupling by an order of magnitude
compared with the current best constraints.
