  In this paper, we investigate the instability in the standard dense retrieval
training, which iterates between model training and hard negative selection
using the being-trained model. We show the catastrophic forgetting phenomena
behind the training instability, where models learn and forget different
negative groups during training iterations. We then propose ANCE-Tele, which
accumulates momentum negatives from past iterations and approximates future
iterations using lookahead negatives, as "teleportations" along the time axis
to smooth the learning process. On web search and OpenQA, ANCE-Tele outperforms
previous state-of-the-art systems of similar size, eliminates the dependency on
sparse retrieval negatives, and is competitive among systems using
significantly more (50x) parameters. Our analysis demonstrates that
teleportation negatives reduce catastrophic forgetting and improve convergence
speed for dense retrieval training. Our code is available at
https://github.com/OpenMatch/ANCE-Tele.
