  We study the impact of gravitational wave memory on the distribution of far
away light sources in the sky. For the first time we compute the built up of
small, but permanent tensor distortions of the metric over cosmological
time-scales using realistic models of compact binary coalescences (CBCs) whose
rate of occurrence is extrapolated at $z\sim {\cal O}(1)$. This allows for a
consistent computation of the random-walk like evolution of gravitational wave
memory which, in turn, is used to estimate the overall shape and magnitude of
astrometric deflections of far away sources of light. We find that for pulsar
or quasar proper motions, the near-Earth contribution to the astrometric
deflections dominates the result and the deflection is analogous to a
stochastic gravitational wave memory background that is generally subdominant
to the primary stochastic gravitational wave background. We find that this
contribution can be within the reach of future surveys such as Theia. Finally,
we also study the deviation of the presently observed angular distribution of
quasars from perfect isotropy, which arises from the slow build-up of
gravitational wave memory over the entire history of the universe. In this
case, we find that astrometric deflections depend on the entire light
trajectory from the source to the Earth, yielding a quadruple pattern whose
magnitude is unlikely to be within reach of the next generation of astrometric
surveys due to shot noise and cosmic variance limitations.
